You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "andrijapanicsb (via GitHub)" <gi...@apache.org> on 2026/09/22 10:47:55 UTC
[PR] CKS: reconcile node capacity after live resize [cloudstack]
andrijapanicsb opened a new pull request, #14223:
URL: https://github.com/apache/cloudstack/pull/14223
## Summary
- cordon one Cloud-managed KVM Kubernetes node before a CPU/RAM live resize
- verify guest resources, restart kubelet, and validate Node capacity/allocatable from JSON
- preserve existing cordons and retain a CKS cordon after post-resize verification failure
- resolve per-VM SSH port-forwarding instead of assuming port 2222
## Validation
- mvn -pl plugins/integrations/kubernetes-service -am -DskipTests compile
- focused worker/reconciler tests: 8 passing (-Dexec.skip=true is required on this Windows host to bypass an unrelated Unix shell test in cloud-engine-schema)
## Remaining before ready-for-review
- KVM integration evidence and a QA deployment
- broader failure/order/retry test coverage and release documentation
- dedicated etcd live resize is deliberately rejected until etcd-health reconciliation is added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6470649673
code looks very clean now @andrija, thnks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (WIP)CKS: reconcile node capacity after live resize [cloudstack]
Posted by "andrijapanicsb (via GitHub)" <gi...@apache.org>.
andrijapanicsb commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6322884837
@ACSHomeBot package G
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "ACSHomeBot (via GitHub) via commits" <co...@cloudstack.apache.org>.
ACSHomeBot commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6546837551
Accepted: `package kvm G` for `a8a500f02238`.
Planned: **RPM + DEB**, **KVM SystemVM**.
<!-- acshomebot-package-state:20261031T173610Z-0fbb49d0 -->
Status: **Queued** for **worker G**. Jobs ahead: **0**.
<!-- /acshomebot-package-state:20261031T173610Z-0fbb49d0 -->
<!-- acshomebot-package-status:20261031T173610Z-0fbb49d0 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (WIP)CKS: reconcile node capacity after live resize [cloudstack]
Posted by "andrijapanicsb (via GitHub)" <gi...@apache.org>.
andrijapanicsb commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-5775843850
@ACSHomeBot package G
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "sonarqubecloud[bot] (via GitHub) via commits" <co...@cloudstack.apache.org>.
sonarqubecloud[bot] commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6702275828
## [](https://sonarcloud.io/dashboard?id=apache_cloudstack&pullRequest=14223) **Quality Gate passed**
Issues
 [3 New issues](https://sonarcloud.io/project/issues?id=apache_cloudstack&pullRequest=14223&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_cloudstack&pullRequest=14223&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_cloudstack&pullRequest=14223&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [71.8% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_cloudstack&pullRequest=14223&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_cloudstack&pullRequest=14223&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_cloudstack&pullRequest=14223)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "blueorangutan (via GitHub) via commits" <co...@cloudstack.apache.org>.
blueorangutan commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6730475144
<b>[SF] Trillian test result (tid-17500)</b>
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 55910 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr14223-t17500-kvm-ol8.zip
Smoke tests completed. 157 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (WIP)CKS: reconcile node capacity after live resize [cloudstack]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6284938576
## [](https://sonarcloud.io/dashboard?id=apache_cloudstack&pullRequest=14223) **Quality Gate failed**
Failed conditions
 [31.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_cloudstack&pullRequest=14223&metric=new_coverage&view=list) (required ≥ 40%)
<!-- sqra:qg_revision=5806c81d0faeb37449d67d83c8ea2a985fde189ef1e96055b26315ae6416e529 -->
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_cloudstack&pullRequest=14223)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (WIP)CKS: reconcile node capacity after live resize [cloudstack]
Posted by "ACSHomeBot (via GitHub)" <gi...@apache.org>.
ACSHomeBot commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6322914693
Package building started on **worker G**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "andrijapanicsb (via GitHub)" <gi...@apache.org>.
andrijapanicsb commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6469557498
@DaanHoogland, unrelated to all this work - share with @rajujith some of the PR fixes that I've raised against virt-v2v that are critical for VMware migrations, I'm sure Jithin will appreciate those - https://github.com/libguestfs/virt-v2v/pulls?q=is%3Apr+author%3Aandrijapanicsb - all merged, and he can just take the source DEB/RPM (whatever your worker uses) and backport those, together with the stupid FSTRIM that virt-v2v does - I had a 30TB migration over weekened, and FSTRIM added 2h of duration inside virt-v2v on top of 200 seconds that it really took to convert a WIndows VM with 12 disks/40TB of space. Perhaps he wants to also backport the --no-fstrim option from upstream.
@rajujith ping me if you want to test openvddk.com - it's NBD/NBDSSL and HotAdd - that is a 1:1 binary compatible replacement for VMware's VDDK - completely free, I'm sure you might want to test it in your inhouse product.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "andrijapanicsb (via GitHub)" <gi...@apache.org>.
andrijapanicsb commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6470967713
> > that is a 1:1 binary compatible replacement for VMware's VDDK
>
> @andrijapanicsb , looks like a compatitor of the one mentioned in https://www.networkworld.com/article/4241265/fluidcloud-replaces-vddk-with-its-own-disk-copy-engine-for-vmware-exits.html
>
> needs a comparison? (in a discussion item, probably not here)
@DaanHoogland you know me well enough - don't compare apples and oranges... ;)
"The FluidCloud engine exports the VM disk using Network File Copy **(NFC)** through standard application programming interfaces (API) on the VMware ESXi hypervisor. It then streams the virtual machine disk (VMDK) file over the network to the destination. Omar said the export captures everything, with no functionality loss.
**The engine does not do delta sync, so every export is a full export of the VM.**"
Their stuff is a workaorund but uses NFS, My artefact is the libvixdisk.so.8.0.3 that you replace and off you go (HotAdd, NBD/NBDDSSL (via NFS) - 100% compatible with VDDK = so all tools previously designed to work with VMware VDDK now works with OpenVDDK. check the sites, numbers, compatibilityy etc. I will share this publicly, soon. Free and opensource.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (WIP)CKS: reconcile node capacity after live resize [cloudstack]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-5776714302
## [](https://sonarcloud.io/dashboard?id=apache_cloudstack&pullRequest=14223) **Quality Gate failed**
Failed conditions
 [28.9% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_cloudstack&pullRequest=14223&metric=new_coverage&view=list) (required ≥ 40%)
<!-- sqra:qg_revision=b21a6ce495fd81d2cbc417e4fa9d19786fb3d21b19b321f0ccda97d4e462a9fc -->
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_cloudstack&pullRequest=14223)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "andrijapanicsb (via GitHub)" <gi...@apache.org>.
andrijapanicsb commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6326596856
@blueorangutan package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "blueorangutan (via GitHub) via commits" <co...@cloudstack.apache.org>.
blueorangutan commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6719829727
@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "blueorangutan (via GitHub) via commits" <co...@cloudstack.apache.org>.
blueorangutan commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6706972223
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19931
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "andrijapanicsb (via GitHub)" <gi...@apache.org>.
andrijapanicsb commented on code in PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#discussion_r4394971237
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterScaleWorker.java:
##########
@@ -376,24 +377,171 @@ private void scaleKubernetesClusterOffering(KubernetesClusterNodeType nodeType,
List<KubernetesClusterVmMapVO> vmList = kubernetesClusterVmMapDao.listByClusterIdAndVmType(kubernetesCluster.getId(), nodeType);
final long tobeScaledVMCount = Math.min(vmList.size(), size);
for (long i = 0; i < tobeScaledVMCount; i++) {
- KubernetesClusterVmMapVO vmMapVO = vmList.get((int) i);
- UserVmVO userVM = userVmDao.findById(vmMapVO.getVmId());
- boolean result = false;
- try {
- result = userVmManager.upgradeVirtualMachine(userVM.getId(), serviceOffering.getId(), new HashMap<String, String>());
- } catch (RuntimeException | ResourceUnavailableException | ManagementServerException | VirtualMachineMigrationException e) {
- logTransitStateAndThrow(Level.ERROR, String.format("Scaling Kubernetes cluster : %s failed, unable to scale cluster VM : %s due to %s", kubernetesCluster.getName(), userVM.getDisplayName(), e.getMessage()), kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed, e);
- }
- if (!result) {
- logTransitStateAndThrow(Level.WARN, String.format("Scaling Kubernetes cluster : %s failed, unable to scale cluster VM : %s", kubernetesCluster.getName(), userVM.getDisplayName()),kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
- }
+ new NodeOfferingResize(vmList.get((int) i), nodeType, serviceOffering).execute();
if (System.currentTimeMillis() > scaleTimeoutTime) {
logTransitStateAndThrow(Level.WARN, String.format("Scaling Kubernetes cluster : %s failed, scaling action timed out", kubernetesCluster.getName()),kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
}
}
kubernetesCluster = updateKubernetesClusterEntryForNodeType(null, nodeType, serviceOffering, updateNodeOffering, updateClusterOffering);
}
+ private final class NodeOfferingResize {
Review Comment:
Done, it's now `KubernetesClusterNodeOfferingResizer`, with only the dependencies it needs.
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterScaleWorker.java:
##########
@@ -376,24 +377,171 @@ private void scaleKubernetesClusterOffering(KubernetesClusterNodeType nodeType,
List<KubernetesClusterVmMapVO> vmList = kubernetesClusterVmMapDao.listByClusterIdAndVmType(kubernetesCluster.getId(), nodeType);
final long tobeScaledVMCount = Math.min(vmList.size(), size);
for (long i = 0; i < tobeScaledVMCount; i++) {
- KubernetesClusterVmMapVO vmMapVO = vmList.get((int) i);
- UserVmVO userVM = userVmDao.findById(vmMapVO.getVmId());
- boolean result = false;
- try {
- result = userVmManager.upgradeVirtualMachine(userVM.getId(), serviceOffering.getId(), new HashMap<String, String>());
- } catch (RuntimeException | ResourceUnavailableException | ManagementServerException | VirtualMachineMigrationException e) {
- logTransitStateAndThrow(Level.ERROR, String.format("Scaling Kubernetes cluster : %s failed, unable to scale cluster VM : %s due to %s", kubernetesCluster.getName(), userVM.getDisplayName(), e.getMessage()), kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed, e);
- }
- if (!result) {
- logTransitStateAndThrow(Level.WARN, String.format("Scaling Kubernetes cluster : %s failed, unable to scale cluster VM : %s", kubernetesCluster.getName(), userVM.getDisplayName()),kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
- }
+ new NodeOfferingResize(vmList.get((int) i), nodeType, serviceOffering).execute();
if (System.currentTimeMillis() > scaleTimeoutTime) {
logTransitStateAndThrow(Level.WARN, String.format("Scaling Kubernetes cluster : %s failed, scaling action timed out", kubernetesCluster.getName()),kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
}
}
kubernetesCluster = updateKubernetesClusterEntryForNodeType(null, nodeType, serviceOffering, updateNodeOffering, updateClusterOffering);
}
+ private final class NodeOfferingResize {
+ private final KubernetesClusterVmMapVO vmMap;
+ private final KubernetesClusterNodeType nodeType;
+ private final ServiceOffering targetOffering;
+ private final ServiceOffering oldOffering;
+ private UserVmVO vm;
+ private NodeAccess access;
+ private NodeCapacitySnapshot before;
+ private boolean reconcileKubernetes;
+ private boolean cksCordonCompleted;
+ private boolean resizeSucceeded;
+
+ private NodeOfferingResize(KubernetesClusterVmMapVO vmMap, KubernetesClusterNodeType nodeType, ServiceOffering targetOffering) {
+ this.vmMap = vmMap;
+ this.nodeType = nodeType;
+ this.targetOffering = targetOffering;
+ vm = userVmDao.findById(vmMap.getVmId());
+ if (vm == null) {
+ logTransitStateAndThrow(Level.ERROR, String.format("Scaling Kubernetes cluster : %s failed, unable to find cluster VM %s",
+ kubernetesCluster.getName(), vmMap.getVmId()), kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
+ }
+ oldOffering = serviceOfferingDao.findById(vm.getServiceOfferingId());
+ resizeSucceeded = Objects.equals(targetOffering.getId(), vm.getServiceOfferingId());
+ }
+
+ private void execute() {
+ validateLiveResize();
+ try {
+ prepareReconciliation();
+ resizeVm();
+ reconcileCapacity();
+ } catch (ResourceUnavailableException | ManagementServerException | VirtualMachineMigrationException | CloudRuntimeException e) {
+ restoreAfterFailedResize();
+ reportFailure(e);
+ }
+ }
+
+ private void validateLiveResize() {
+ if (KubernetesClusterNodeCapacityReconcilerImpl.capacityChanged(oldOffering, targetOffering)
+ && KubernetesCluster.State.Running == originalState && ETCD == nodeType
+ && KubernetesCluster.ClusterType.CloudManaged == kubernetesCluster.getClusterType()) {
+ logTransitStateAndThrow(Level.ERROR, String.format("Scaling Kubernetes cluster : %s cannot live-resize dedicated etcd VM %s; " +
+ "etcd health reconciliation is not implemented", kubernetesCluster.getName(), vm.getDisplayName()),
+ kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
+ }
Review Comment:
Thanks for catching this. Looking at it again, the check itself was the problem. On main, dedicated etcd VMs are live-resized like the other VMs, and since they aren't Kubernetes nodes there's nothing to reconcile on them. So I removed the check: etcd VMs get exactly the same resize as on main, and a request that includes etcd is no longer rejected halfway through.
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterScaleWorker.java:
##########
@@ -376,24 +377,171 @@ private void scaleKubernetesClusterOffering(KubernetesClusterNodeType nodeType,
List<KubernetesClusterVmMapVO> vmList = kubernetesClusterVmMapDao.listByClusterIdAndVmType(kubernetesCluster.getId(), nodeType);
final long tobeScaledVMCount = Math.min(vmList.size(), size);
for (long i = 0; i < tobeScaledVMCount; i++) {
- KubernetesClusterVmMapVO vmMapVO = vmList.get((int) i);
- UserVmVO userVM = userVmDao.findById(vmMapVO.getVmId());
- boolean result = false;
- try {
- result = userVmManager.upgradeVirtualMachine(userVM.getId(), serviceOffering.getId(), new HashMap<String, String>());
- } catch (RuntimeException | ResourceUnavailableException | ManagementServerException | VirtualMachineMigrationException e) {
- logTransitStateAndThrow(Level.ERROR, String.format("Scaling Kubernetes cluster : %s failed, unable to scale cluster VM : %s due to %s", kubernetesCluster.getName(), userVM.getDisplayName(), e.getMessage()), kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed, e);
- }
- if (!result) {
- logTransitStateAndThrow(Level.WARN, String.format("Scaling Kubernetes cluster : %s failed, unable to scale cluster VM : %s", kubernetesCluster.getName(), userVM.getDisplayName()),kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
- }
+ new NodeOfferingResize(vmList.get((int) i), nodeType, serviceOffering).execute();
if (System.currentTimeMillis() > scaleTimeoutTime) {
logTransitStateAndThrow(Level.WARN, String.format("Scaling Kubernetes cluster : %s failed, scaling action timed out", kubernetesCluster.getName()),kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
}
}
kubernetesCluster = updateKubernetesClusterEntryForNodeType(null, nodeType, serviceOffering, updateNodeOffering, updateClusterOffering);
}
+ private final class NodeOfferingResize {
+ private final KubernetesClusterVmMapVO vmMap;
+ private final KubernetesClusterNodeType nodeType;
+ private final ServiceOffering targetOffering;
+ private final ServiceOffering oldOffering;
+ private UserVmVO vm;
+ private NodeAccess access;
+ private NodeCapacitySnapshot before;
+ private boolean reconcileKubernetes;
+ private boolean cksCordonCompleted;
+ private boolean resizeSucceeded;
+
+ private NodeOfferingResize(KubernetesClusterVmMapVO vmMap, KubernetesClusterNodeType nodeType, ServiceOffering targetOffering) {
+ this.vmMap = vmMap;
+ this.nodeType = nodeType;
+ this.targetOffering = targetOffering;
+ vm = userVmDao.findById(vmMap.getVmId());
+ if (vm == null) {
+ logTransitStateAndThrow(Level.ERROR, String.format("Scaling Kubernetes cluster : %s failed, unable to find cluster VM %s",
+ kubernetesCluster.getName(), vmMap.getVmId()), kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
+ }
+ oldOffering = serviceOfferingDao.findById(vm.getServiceOfferingId());
+ resizeSucceeded = Objects.equals(targetOffering.getId(), vm.getServiceOfferingId());
+ }
+
+ private void execute() {
+ validateLiveResize();
+ try {
+ prepareReconciliation();
+ resizeVm();
+ reconcileCapacity();
+ } catch (ResourceUnavailableException | ManagementServerException | VirtualMachineMigrationException | CloudRuntimeException e) {
+ restoreAfterFailedResize();
+ reportFailure(e);
+ }
+ }
+
+ private void validateLiveResize() {
+ if (KubernetesClusterNodeCapacityReconcilerImpl.capacityChanged(oldOffering, targetOffering)
+ && KubernetesCluster.State.Running == originalState && ETCD == nodeType
+ && KubernetesCluster.ClusterType.CloudManaged == kubernetesCluster.getClusterType()) {
+ logTransitStateAndThrow(Level.ERROR, String.format("Scaling Kubernetes cluster : %s cannot live-resize dedicated etcd VM %s; " +
+ "etcd health reconciliation is not implemented", kubernetesCluster.getName(), vm.getDisplayName()),
+ kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
+ }
+ }
+
+ private void prepareReconciliation() {
+ reconcileKubernetes = shouldReconcileNodeCapacity(vmMap, vm, oldOffering, targetOffering, nodeType);
Review Comment:
Done, there's one log line per node with the reason (external node, not CloudManaged, not KVM, cluster not Running, not a control or worker VM, no CPU/RAM change, or already on the target offering).
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterScaleWorker.java:
##########
@@ -376,24 +377,171 @@ private void scaleKubernetesClusterOffering(KubernetesClusterNodeType nodeType,
List<KubernetesClusterVmMapVO> vmList = kubernetesClusterVmMapDao.listByClusterIdAndVmType(kubernetesCluster.getId(), nodeType);
final long tobeScaledVMCount = Math.min(vmList.size(), size);
for (long i = 0; i < tobeScaledVMCount; i++) {
- KubernetesClusterVmMapVO vmMapVO = vmList.get((int) i);
- UserVmVO userVM = userVmDao.findById(vmMapVO.getVmId());
- boolean result = false;
- try {
- result = userVmManager.upgradeVirtualMachine(userVM.getId(), serviceOffering.getId(), new HashMap<String, String>());
- } catch (RuntimeException | ResourceUnavailableException | ManagementServerException | VirtualMachineMigrationException e) {
- logTransitStateAndThrow(Level.ERROR, String.format("Scaling Kubernetes cluster : %s failed, unable to scale cluster VM : %s due to %s", kubernetesCluster.getName(), userVM.getDisplayName(), e.getMessage()), kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed, e);
- }
- if (!result) {
- logTransitStateAndThrow(Level.WARN, String.format("Scaling Kubernetes cluster : %s failed, unable to scale cluster VM : %s", kubernetesCluster.getName(), userVM.getDisplayName()),kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
- }
+ new NodeOfferingResize(vmList.get((int) i), nodeType, serviceOffering).execute();
if (System.currentTimeMillis() > scaleTimeoutTime) {
logTransitStateAndThrow(Level.WARN, String.format("Scaling Kubernetes cluster : %s failed, scaling action timed out", kubernetesCluster.getName()),kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
}
}
kubernetesCluster = updateKubernetesClusterEntryForNodeType(null, nodeType, serviceOffering, updateNodeOffering, updateClusterOffering);
}
+ private final class NodeOfferingResize {
+ private final KubernetesClusterVmMapVO vmMap;
+ private final KubernetesClusterNodeType nodeType;
+ private final ServiceOffering targetOffering;
+ private final ServiceOffering oldOffering;
+ private UserVmVO vm;
+ private NodeAccess access;
+ private NodeCapacitySnapshot before;
+ private boolean reconcileKubernetes;
+ private boolean cksCordonCompleted;
+ private boolean resizeSucceeded;
+
+ private NodeOfferingResize(KubernetesClusterVmMapVO vmMap, KubernetesClusterNodeType nodeType, ServiceOffering targetOffering) {
+ this.vmMap = vmMap;
+ this.nodeType = nodeType;
+ this.targetOffering = targetOffering;
+ vm = userVmDao.findById(vmMap.getVmId());
+ if (vm == null) {
+ logTransitStateAndThrow(Level.ERROR, String.format("Scaling Kubernetes cluster : %s failed, unable to find cluster VM %s",
+ kubernetesCluster.getName(), vmMap.getVmId()), kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
+ }
+ oldOffering = serviceOfferingDao.findById(vm.getServiceOfferingId());
+ resizeSucceeded = Objects.equals(targetOffering.getId(), vm.getServiceOfferingId());
+ }
+
+ private void execute() {
+ validateLiveResize();
+ try {
+ prepareReconciliation();
+ resizeVm();
+ reconcileCapacity();
+ } catch (ResourceUnavailableException | ManagementServerException | VirtualMachineMigrationException | CloudRuntimeException e) {
+ restoreAfterFailedResize();
+ reportFailure(e);
+ }
+ }
+
+ private void validateLiveResize() {
+ if (KubernetesClusterNodeCapacityReconcilerImpl.capacityChanged(oldOffering, targetOffering)
+ && KubernetesCluster.State.Running == originalState && ETCD == nodeType
+ && KubernetesCluster.ClusterType.CloudManaged == kubernetesCluster.getClusterType()) {
+ logTransitStateAndThrow(Level.ERROR, String.format("Scaling Kubernetes cluster : %s cannot live-resize dedicated etcd VM %s; " +
+ "etcd health reconciliation is not implemented", kubernetesCluster.getName(), vm.getDisplayName()),
+ kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
+ }
+ }
+
+ private void prepareReconciliation() {
+ reconcileKubernetes = shouldReconcileNodeCapacity(vmMap, vm, oldOffering, targetOffering, nodeType);
+ if (reconcileKubernetes || isReconciliationRetryCandidate(vmMap, vm, targetOffering, nodeType)) {
+ access = resolveNodeAccess(vm);
+ before = kubernetesClusterNodeCapacityReconciler.captureBefore(kubernetesCluster, vm, access);
Review Comment:
Agreed that it shouldn't need SSH to the node. I didn't want to skip on an error though: if that node still had unfinished work, skipping would leave it with the old capacity and nobody would notice. So I changed how it's detected. The node now gets a small "pending" annotation when its resize starts, and it's removed only once the node is verified. For a node that's already on the target offering, only the Node object is read through the control plane: no annotation means it's skipped without any SSH to the node, and with the annotation it's finished first. This also fixed a case I had missed, where a node cordoned by an operator would never have been retried.
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterScaleWorker.java:
##########
@@ -376,24 +377,171 @@ private void scaleKubernetesClusterOffering(KubernetesClusterNodeType nodeType,
List<KubernetesClusterVmMapVO> vmList = kubernetesClusterVmMapDao.listByClusterIdAndVmType(kubernetesCluster.getId(), nodeType);
final long tobeScaledVMCount = Math.min(vmList.size(), size);
for (long i = 0; i < tobeScaledVMCount; i++) {
- KubernetesClusterVmMapVO vmMapVO = vmList.get((int) i);
- UserVmVO userVM = userVmDao.findById(vmMapVO.getVmId());
- boolean result = false;
- try {
- result = userVmManager.upgradeVirtualMachine(userVM.getId(), serviceOffering.getId(), new HashMap<String, String>());
- } catch (RuntimeException | ResourceUnavailableException | ManagementServerException | VirtualMachineMigrationException e) {
- logTransitStateAndThrow(Level.ERROR, String.format("Scaling Kubernetes cluster : %s failed, unable to scale cluster VM : %s due to %s", kubernetesCluster.getName(), userVM.getDisplayName(), e.getMessage()), kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed, e);
- }
- if (!result) {
- logTransitStateAndThrow(Level.WARN, String.format("Scaling Kubernetes cluster : %s failed, unable to scale cluster VM : %s", kubernetesCluster.getName(), userVM.getDisplayName()),kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
- }
+ new NodeOfferingResize(vmList.get((int) i), nodeType, serviceOffering).execute();
if (System.currentTimeMillis() > scaleTimeoutTime) {
logTransitStateAndThrow(Level.WARN, String.format("Scaling Kubernetes cluster : %s failed, scaling action timed out", kubernetesCluster.getName()),kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
}
}
kubernetesCluster = updateKubernetesClusterEntryForNodeType(null, nodeType, serviceOffering, updateNodeOffering, updateClusterOffering);
}
+ private final class NodeOfferingResize {
+ private final KubernetesClusterVmMapVO vmMap;
+ private final KubernetesClusterNodeType nodeType;
+ private final ServiceOffering targetOffering;
+ private final ServiceOffering oldOffering;
+ private UserVmVO vm;
+ private NodeAccess access;
+ private NodeCapacitySnapshot before;
+ private boolean reconcileKubernetes;
+ private boolean cksCordonCompleted;
+ private boolean resizeSucceeded;
+
+ private NodeOfferingResize(KubernetesClusterVmMapVO vmMap, KubernetesClusterNodeType nodeType, ServiceOffering targetOffering) {
+ this.vmMap = vmMap;
+ this.nodeType = nodeType;
+ this.targetOffering = targetOffering;
+ vm = userVmDao.findById(vmMap.getVmId());
+ if (vm == null) {
+ logTransitStateAndThrow(Level.ERROR, String.format("Scaling Kubernetes cluster : %s failed, unable to find cluster VM %s",
+ kubernetesCluster.getName(), vmMap.getVmId()), kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
+ }
+ oldOffering = serviceOfferingDao.findById(vm.getServiceOfferingId());
+ resizeSucceeded = Objects.equals(targetOffering.getId(), vm.getServiceOfferingId());
+ }
+
+ private void execute() {
+ validateLiveResize();
+ try {
+ prepareReconciliation();
+ resizeVm();
+ reconcileCapacity();
+ } catch (ResourceUnavailableException | ManagementServerException | VirtualMachineMigrationException | CloudRuntimeException e) {
+ restoreAfterFailedResize();
+ reportFailure(e);
+ }
+ }
+
+ private void validateLiveResize() {
+ if (KubernetesClusterNodeCapacityReconcilerImpl.capacityChanged(oldOffering, targetOffering)
+ && KubernetesCluster.State.Running == originalState && ETCD == nodeType
+ && KubernetesCluster.ClusterType.CloudManaged == kubernetesCluster.getClusterType()) {
+ logTransitStateAndThrow(Level.ERROR, String.format("Scaling Kubernetes cluster : %s cannot live-resize dedicated etcd VM %s; " +
+ "etcd health reconciliation is not implemented", kubernetesCluster.getName(), vm.getDisplayName()),
+ kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
+ }
+ }
+
+ private void prepareReconciliation() {
+ reconcileKubernetes = shouldReconcileNodeCapacity(vmMap, vm, oldOffering, targetOffering, nodeType);
+ if (reconcileKubernetes || isReconciliationRetryCandidate(vmMap, vm, targetOffering, nodeType)) {
+ access = resolveNodeAccess(vm);
+ before = kubernetesClusterNodeCapacityReconciler.captureBefore(kubernetesCluster, vm, access);
+ reconcileKubernetes = reconcileKubernetes || before.isCloudStackResizeCordon();
+ }
+ if (reconcileKubernetes) {
+ kubernetesClusterNodeCapacityReconciler.cordonIfNeeded(kubernetesCluster, vm, before, access, scaleTimeoutTime);
+ cksCordonCompleted = !before.isUnschedulable() || before.isCloudStackResizeCordon();
+ }
+ }
+
+ private void resizeVm() throws ResourceUnavailableException, ManagementServerException, VirtualMachineMigrationException {
+ if (resizeSucceeded) {
+ return;
+ }
+ if (!userVmManager.upgradeVirtualMachine(vm.getId(), targetOffering.getId(), new HashMap<>())) {
+ throw new CloudRuntimeException("Unable to scale cluster VM " + vm.getDisplayName());
+ }
+ resizeSucceeded = true;
+ UserVmVO resizedVm = userVmDao.findById(vm.getId());
+ if (resizedVm == null || !Objects.equals(targetOffering.getId(), resizedVm.getServiceOfferingId())) {
+ throw new CloudRuntimeException("VM " + vm.getUuid() + " did not reach target offering");
+ }
+ vm = resizedVm;
+ }
+
+ private void reconcileCapacity() {
+ if (!reconcileKubernetes) {
+ return;
+ }
+ kubernetesClusterNodeCapacityReconciler.onlineGuestResources(access);
+ kubernetesClusterNodeCapacityReconciler.verifyGuestResources(vm, targetOffering, before, access, scaleTimeoutTime);
+ if (!kubernetesClusterNodeCapacityReconciler.isKubernetesResourcesCurrent(before, targetOffering)) {
+ kubernetesClusterNodeCapacityReconciler.restartKubelet(vm, access, scaleTimeoutTime);
+ kubernetesClusterNodeCapacityReconciler.waitForKubernetesResources(kubernetesCluster, vm, targetOffering, before, access, scaleTimeoutTime);
+ }
+ kubernetesClusterNodeCapacityReconciler.restoreSchedulability(kubernetesCluster, vm, before, access, scaleTimeoutTime);
+ }
+
+ private void restoreAfterFailedResize() {
+ if (!reconcileKubernetes || !cksCordonCompleted || resizeSucceeded) {
+ return;
+ }
+ try {
+ kubernetesClusterNodeCapacityReconciler.restoreSchedulability(kubernetesCluster, vm, before, access, scaleTimeoutTime);
+ } catch (CloudRuntimeException cleanupException) {
+ logger.warn("Unable to restore schedulability for CKS node {} after a pre-resize failure", vm.getUuid(), cleanupException);
+ }
Review Comment:
You're right. Two changes: errors from parsing the `kubectl` and guest output are now turned into `NodeCapacityReconciliationException` inside the reconciler, with the phase and the VM in the message. And the per-node handler catches `RuntimeException` again together with the three checked exceptions, the same list as on main before this PR, so an unexpected error still goes through the cordon cleanup and marks the scale as failed. The original exception is kept as the cause and logged with its stack trace. I also replaced the test that expected the old behaviour.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6470512737
## [](https://sonarcloud.io/dashboard?id=apache_cloudstack&pullRequest=14223) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_cloudstack&pullRequest=14223&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_cloudstack&pullRequest=14223&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_cloudstack&pullRequest=14223&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [72.7% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_cloudstack&pullRequest=14223&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_cloudstack&pullRequest=14223&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_cloudstack&pullRequest=14223)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "andrijapanicsb (via GitHub)" <gi...@apache.org>.
andrijapanicsb commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6470985920
@ACSHomeBot package G
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "andrijapanicsb (via GitHub) via commits" <co...@cloudstack.apache.org>.
andrijapanicsb commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6546835819
@ACSHomeBot package G
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (WIP)CKS: reconcile node capacity after live resize [cloudstack]
Posted by "ACSHomeBot (via GitHub)" <gi...@apache.org>.
ACSHomeBot commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6324030729
### Packaging results:
| Result | Artifact | Platform | Build Time |
|:--|:--|:--|--:|
| PASS | RPM | EL (EL8/9/10) | 30 min |
| PASS | DEB | Ubuntu, Debian | 22 min |
**Test packages are available at:**
- EL9/RPM: [https://packages.v2kmigrate.com/cloudstack/pr/14223/el9/](https://packages.v2kmigrate.com/cloudstack/pr/14223/el9/index.html)
- Debian/APT: [https://packages.v2kmigrate.com/cloudstack/pr/14223/debian/](https://packages.v2kmigrate.com/cloudstack/pr/14223/debian/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6411322406
@blueorangutan test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on code in PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#discussion_r4375191079
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterScaleWorker.java:
##########
@@ -361,7 +361,7 @@ private void validateKubernetesClusterScaleSizeParameters(KubernetesClusterNodeT
}
}
- private void scaleKubernetesClusterOffering(KubernetesClusterNodeType nodeType, ServiceOffering serviceOffering,
+ protected void scaleKubernetesClusterOffering(KubernetesClusterNodeType nodeType, ServiceOffering serviceOffering,
Review Comment:
@andrijapanicsb , please deal with the sonar warning here and revert the catch all further on in the method. I'd suggest you instruct your AI to use 'clean coding principles' as standard skill.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "ACSHomeBot (via GitHub)" <gi...@apache.org>.
ACSHomeBot commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6470999216
Accepted: `package kvm G` for `c48bd344eeb1`.
Planned: **RPM + DEB**, **KVM SystemVM**.
<!-- acshomebot-package-state:20261028T135941Z-787a23b0 -->
Status: **Queued** for **worker G**. Jobs ahead: **1**.
<!-- /acshomebot-package-state:20261028T135941Z-787a23b0 -->
<!-- acshomebot-package-status:20261028T135941Z-787a23b0 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "ACSHomeBot (via GitHub)" <gi...@apache.org>.
ACSHomeBot commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6471624587
### Packaging results:
| Result | Artifact | Platform | Build Time |
|:--|:--|:--|--:|
| PASS | RPM | EL (EL8/9/10) | 34 min |
| PASS | DEB | Ubuntu, Debian | 23 min |
**Test packages are available at:**
- EL9/RPM: [https://packages.v2kmigrate.com/cloudstack/pr/14223/el9/](https://packages.v2kmigrate.com/cloudstack/pr/14223/el9/index.html)
- Debian/APT: [https://packages.v2kmigrate.com/cloudstack/pr/14223/debian/](https://packages.v2kmigrate.com/cloudstack/pr/14223/debian/index.html)
<!-- acshomebot-package-results:20261028T132312Z-adeb3d74:1 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "blueorangutan (via GitHub) via commits" <co...@cloudstack.apache.org>.
blueorangutan commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6568256817
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✖️ suse15. SL-JID 19872
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6438585000
> The technical findings are fair, and I’ve addressed them. The remark about instructing “my AI”, however, is unnecessary. Please keep the review focused on concrete issues in the code.
ok, then please keep to clean code principles for future submissions. (This is not personally to you Andrija and stating things like this is very necessary in this project. Also not personally to you. I do not know what AI you use to generate your code, hence "your" AI!)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "DaanHoogland (via GitHub) via commits" <co...@cloudstack.apache.org>.
DaanHoogland commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6658063482
> suse build failed:
>
> ```
> 02:52:41 [ERROR] Failures:
> 02:52:41 [ERROR] MetalinkDirectTemplateDownloaderTest.testCheckUrlMetalink:33
> ```
>
> looks like a generic test so rebuilding in the background
false positive, i.e. rebuild passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "andrijapanicsb (via GitHub)" <gi...@apache.org>.
andrijapanicsb commented on code in PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#discussion_r4400708403
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterScaleWorker.java:
##########
@@ -361,7 +361,7 @@ private void validateKubernetesClusterScaleSizeParameters(KubernetesClusterNodeT
}
}
- private void scaleKubernetesClusterOffering(KubernetesClusterNodeType nodeType, ServiceOffering serviceOffering,
+ protected void scaleKubernetesClusterOffering(KubernetesClusterNodeType nodeType, ServiceOffering serviceOffering,
Review Comment:
Thanks Daan. Those were the 9 SonarCloud findings, they're fixed in c48bd344ee, and Sonar should clear them once it finishes analysing that commit. The other annotations on the build checks ("TestHook interrupted while sleeping") are log lines from DisconnectHooksTest in the KVM plugin, which this PR doesn't touch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6327893409
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19660
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "andrijapanicsb (via GitHub)" <gi...@apache.org>.
andrijapanicsb commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6468706955
@ACSHomeBot package G
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6523957829
<b>[SF] Trillian test result (tid-17329)</b>
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 64763 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr14223-t17329-kvm-ol8.zip
Smoke tests completed. 157 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (WIP)CKS: reconcile node capacity after live resize [cloudstack]
Posted by "ACSHomeBot (via GitHub)" <gi...@apache.org>.
ACSHomeBot commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6292118234
### Packaging results:
| Result | Artifact | Platform | Build Time |
|:--|:--|:--|--:|
| NOT RUN | RPM | EL (EL8/9/10) | n/a |
| NOT RUN | DEB | Ubuntu, Debian | n/a |
Package building did not start. No repository was published.
Status: **Infrastructure failure before package building started.**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "andrijapanicsb (via GitHub)" <gi...@apache.org>.
andrijapanicsb commented on code in PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#discussion_r4394974429
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/utils/KubernetesClusterNodeCapacityReconcilerImpl.java:
##########
@@ -0,0 +1,232 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.kubernetes.cluster.utils;
+
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import com.cloud.kubernetes.cluster.KubernetesCluster;
+import com.cloud.kubernetes.cluster.KubernetesServiceHelper.KubernetesClusterNodeType;
+import com.cloud.offering.ServiceOffering;
+import com.cloud.uservm.UserVm;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.ssh.SshHelper;
+import com.google.gson.JsonArray;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParser;
+import org.apache.commons.lang3.StringUtils;
+
+/** SSH-backed implementation for a rolling CKS live-resize reconciliation. */
+public class KubernetesClusterNodeCapacityReconcilerImpl implements KubernetesClusterNodeCapacityReconciler {
+ // KVM QA observes 4,568,788 KiB in a 4,608 MiB guest after memory hotplug
+ // (about 146 MiB below the offering), so keep a small margin above that
+ // platform overhead while still rejecting materially undersized guests.
+ static final long MINIMUM_MEMORY_OVERHEAD_BYTES = 160L * 1024L * 1024L;
+ static final long KUBERNETES_MEMORY_REPORTING_TOLERANCE_BYTES = 16L * 1024L * 1024L;
+ private static final int COMMAND_TIMEOUT_MS = 30000;
+ private static final int POLL_INTERVAL_MS = 5000;
+ private static final Pattern NODE_NAME = Pattern.compile("[A-Za-z0-9][A-Za-z0-9.-]{0,252}");
+ private static final Pattern MEMORY_QUANTITY = Pattern.compile("([0-9]+)(Ki|Mi|Gi|Ti|K|M|G|T)?");
+ private static final String RESIZE_ANNOTATION = "cloudstack.apache.org/cks-live-resize";
+ static final String ONLINE_HOTPLUGGED_RESOURCES = "sudo sh -c 'for cpu in /sys/devices/system/cpu/cpu[0-9]*; do "
+ + "[ -e \"$cpu/online\" ] || continue; echo 1 > \"$cpu/online\"; done; "
+ + "for memory in /sys/devices/system/memory/memory[0-9]*; do "
+ + "[ -e \"$memory/online\" ] || continue; echo online > \"$memory/online\"; done'";
+
+ @Override
+ public boolean requiresKubeletRefresh(ServiceOffering oldOffering, ServiceOffering newOffering,
+ KubernetesClusterNodeType nodeType, KubernetesCluster.State originalState) {
+ return KubernetesCluster.State.Running == originalState
+ && (KubernetesClusterNodeType.WORKER == nodeType || KubernetesClusterNodeType.CONTROL == nodeType)
+ && capacityChanged(oldOffering, newOffering);
+ }
+
+ @Override
+ public NodeCapacitySnapshot captureBefore(KubernetesCluster cluster, UserVm vm, NodeAccess access) {
+ return nodeSnapshot(cluster, vm, access, guestResources(access));
+ }
+
+ @Override
+ public void cordonIfNeeded(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "=" + cluster.getUuid() + " --overwrite");
+ executeControl(access, "sudo /opt/bin/kubectl cordon " + node);
+ while (System.currentTimeMillis() < deadline) {
Review Comment:
Done, there's one polling helper now, and each caller still reports its own phase in the error.
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/utils/KubernetesClusterNodeCapacityReconcilerImpl.java:
##########
@@ -0,0 +1,232 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.kubernetes.cluster.utils;
+
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import com.cloud.kubernetes.cluster.KubernetesCluster;
+import com.cloud.kubernetes.cluster.KubernetesServiceHelper.KubernetesClusterNodeType;
+import com.cloud.offering.ServiceOffering;
+import com.cloud.uservm.UserVm;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.ssh.SshHelper;
+import com.google.gson.JsonArray;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParser;
+import org.apache.commons.lang3.StringUtils;
+
+/** SSH-backed implementation for a rolling CKS live-resize reconciliation. */
+public class KubernetesClusterNodeCapacityReconcilerImpl implements KubernetesClusterNodeCapacityReconciler {
+ // KVM QA observes 4,568,788 KiB in a 4,608 MiB guest after memory hotplug
+ // (about 146 MiB below the offering), so keep a small margin above that
+ // platform overhead while still rejecting materially undersized guests.
+ static final long MINIMUM_MEMORY_OVERHEAD_BYTES = 160L * 1024L * 1024L;
+ static final long KUBERNETES_MEMORY_REPORTING_TOLERANCE_BYTES = 16L * 1024L * 1024L;
+ private static final int COMMAND_TIMEOUT_MS = 30000;
+ private static final int POLL_INTERVAL_MS = 5000;
+ private static final Pattern NODE_NAME = Pattern.compile("[A-Za-z0-9][A-Za-z0-9.-]{0,252}");
+ private static final Pattern MEMORY_QUANTITY = Pattern.compile("([0-9]+)(Ki|Mi|Gi|Ti|K|M|G|T)?");
+ private static final String RESIZE_ANNOTATION = "cloudstack.apache.org/cks-live-resize";
+ static final String ONLINE_HOTPLUGGED_RESOURCES = "sudo sh -c 'for cpu in /sys/devices/system/cpu/cpu[0-9]*; do "
+ + "[ -e \"$cpu/online\" ] || continue; echo 1 > \"$cpu/online\"; done; "
+ + "for memory in /sys/devices/system/memory/memory[0-9]*; do "
+ + "[ -e \"$memory/online\" ] || continue; echo online > \"$memory/online\"; done'";
+
+ @Override
+ public boolean requiresKubeletRefresh(ServiceOffering oldOffering, ServiceOffering newOffering,
+ KubernetesClusterNodeType nodeType, KubernetesCluster.State originalState) {
+ return KubernetesCluster.State.Running == originalState
+ && (KubernetesClusterNodeType.WORKER == nodeType || KubernetesClusterNodeType.CONTROL == nodeType)
+ && capacityChanged(oldOffering, newOffering);
+ }
+
+ @Override
+ public NodeCapacitySnapshot captureBefore(KubernetesCluster cluster, UserVm vm, NodeAccess access) {
+ return nodeSnapshot(cluster, vm, access, guestResources(access));
+ }
+
+ @Override
+ public void cordonIfNeeded(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "=" + cluster.getUuid() + " --overwrite");
+ executeControl(access, "sudo /opt/bin/kubectl cordon " + node);
+ while (System.currentTimeMillis() < deadline) {
+ if (nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("CORDON", vm, "Kubernetes node did not become unschedulable");
Review Comment:
I looked at this one closely. The resize needs the annotations and the cordon/uncordon to change in a single update, so a failure can't leave a node half-marked. The existing helper also loses the thread's interrupt flag. So instead of reusing it as is, I added a small helper next to it in `KubernetesClusterUtil` that builds the node patch command, and moved the node-name handling (including the lowercasing) there, so both use it.
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/utils/KubernetesClusterNodeCapacityReconcilerImpl.java:
##########
@@ -0,0 +1,232 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.kubernetes.cluster.utils;
+
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import com.cloud.kubernetes.cluster.KubernetesCluster;
+import com.cloud.kubernetes.cluster.KubernetesServiceHelper.KubernetesClusterNodeType;
+import com.cloud.offering.ServiceOffering;
+import com.cloud.uservm.UserVm;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.ssh.SshHelper;
+import com.google.gson.JsonArray;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParser;
+import org.apache.commons.lang3.StringUtils;
+
+/** SSH-backed implementation for a rolling CKS live-resize reconciliation. */
+public class KubernetesClusterNodeCapacityReconcilerImpl implements KubernetesClusterNodeCapacityReconciler {
+ // KVM QA observes 4,568,788 KiB in a 4,608 MiB guest after memory hotplug
+ // (about 146 MiB below the offering), so keep a small margin above that
+ // platform overhead while still rejecting materially undersized guests.
+ static final long MINIMUM_MEMORY_OVERHEAD_BYTES = 160L * 1024L * 1024L;
+ static final long KUBERNETES_MEMORY_REPORTING_TOLERANCE_BYTES = 16L * 1024L * 1024L;
+ private static final int COMMAND_TIMEOUT_MS = 30000;
+ private static final int POLL_INTERVAL_MS = 5000;
+ private static final Pattern NODE_NAME = Pattern.compile("[A-Za-z0-9][A-Za-z0-9.-]{0,252}");
+ private static final Pattern MEMORY_QUANTITY = Pattern.compile("([0-9]+)(Ki|Mi|Gi|Ti|K|M|G|T)?");
+ private static final String RESIZE_ANNOTATION = "cloudstack.apache.org/cks-live-resize";
+ static final String ONLINE_HOTPLUGGED_RESOURCES = "sudo sh -c 'for cpu in /sys/devices/system/cpu/cpu[0-9]*; do "
+ + "[ -e \"$cpu/online\" ] || continue; echo 1 > \"$cpu/online\"; done; "
+ + "for memory in /sys/devices/system/memory/memory[0-9]*; do "
+ + "[ -e \"$memory/online\" ] || continue; echo online > \"$memory/online\"; done'";
+
+ @Override
+ public boolean requiresKubeletRefresh(ServiceOffering oldOffering, ServiceOffering newOffering,
+ KubernetesClusterNodeType nodeType, KubernetesCluster.State originalState) {
+ return KubernetesCluster.State.Running == originalState
+ && (KubernetesClusterNodeType.WORKER == nodeType || KubernetesClusterNodeType.CONTROL == nodeType)
+ && capacityChanged(oldOffering, newOffering);
+ }
+
+ @Override
+ public NodeCapacitySnapshot captureBefore(KubernetesCluster cluster, UserVm vm, NodeAccess access) {
+ return nodeSnapshot(cluster, vm, access, guestResources(access));
+ }
+
+ @Override
+ public void cordonIfNeeded(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "=" + cluster.getUuid() + " --overwrite");
+ executeControl(access, "sudo /opt/bin/kubectl cordon " + node);
+ while (System.currentTimeMillis() < deadline) {
+ if (nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("CORDON", vm, "Kubernetes node did not become unschedulable");
+ }
+
+ @Override
+ public void onlineGuestResources(NodeAccess access) {
+ executeNode(access, ONLINE_HOTPLUGGED_RESOURCES);
+ }
+
+ @Override
+ public void verifyGuestResources(UserVm vm, ServiceOffering target, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ if (guestMatchesTarget(guestResources(access), target)) return;
+ sleep();
+ }
+ throw failure("GUEST_VERIFY", vm, "guest CPU or memory did not reach the target offering");
+ }
+
+ @Override
+ public boolean isKubernetesResourcesCurrent(NodeCapacitySnapshot snapshot, ServiceOffering target) {
+ GuestResources guest = new GuestResources(snapshot.getGuestOnlineCpuCount(), snapshot.getGuestMemoryKiB());
+ return guestMatchesTarget(guest, target) && kubernetesMatchesGuest(snapshot, snapshot);
Review Comment:
Good catch, the "grew" part meant nothing there, so I removed it. Kubelet computes allocatable as capacity minus the reservations in the same node-status update (the `MachineInfo` setter in `pkg/kubelet/nodestatus/setters.go`), so the check now uses a fresh snapshot: capacity matches the guest, the node is Ready, and allocatable is between 0 and capacity. Kubelet also has to be active before a restart is skipped. Renamed to `isKubernetesNodeAlreadyResized` as you suggested.
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/utils/KubernetesClusterNodeCapacityReconcilerImpl.java:
##########
@@ -0,0 +1,232 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.kubernetes.cluster.utils;
+
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import com.cloud.kubernetes.cluster.KubernetesCluster;
+import com.cloud.kubernetes.cluster.KubernetesServiceHelper.KubernetesClusterNodeType;
+import com.cloud.offering.ServiceOffering;
+import com.cloud.uservm.UserVm;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.ssh.SshHelper;
+import com.google.gson.JsonArray;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParser;
+import org.apache.commons.lang3.StringUtils;
+
+/** SSH-backed implementation for a rolling CKS live-resize reconciliation. */
+public class KubernetesClusterNodeCapacityReconcilerImpl implements KubernetesClusterNodeCapacityReconciler {
+ // KVM QA observes 4,568,788 KiB in a 4,608 MiB guest after memory hotplug
+ // (about 146 MiB below the offering), so keep a small margin above that
+ // platform overhead while still rejecting materially undersized guests.
+ static final long MINIMUM_MEMORY_OVERHEAD_BYTES = 160L * 1024L * 1024L;
+ static final long KUBERNETES_MEMORY_REPORTING_TOLERANCE_BYTES = 16L * 1024L * 1024L;
+ private static final int COMMAND_TIMEOUT_MS = 30000;
+ private static final int POLL_INTERVAL_MS = 5000;
+ private static final Pattern NODE_NAME = Pattern.compile("[A-Za-z0-9][A-Za-z0-9.-]{0,252}");
+ private static final Pattern MEMORY_QUANTITY = Pattern.compile("([0-9]+)(Ki|Mi|Gi|Ti|K|M|G|T)?");
+ private static final String RESIZE_ANNOTATION = "cloudstack.apache.org/cks-live-resize";
+ static final String ONLINE_HOTPLUGGED_RESOURCES = "sudo sh -c 'for cpu in /sys/devices/system/cpu/cpu[0-9]*; do "
+ + "[ -e \"$cpu/online\" ] || continue; echo 1 > \"$cpu/online\"; done; "
+ + "for memory in /sys/devices/system/memory/memory[0-9]*; do "
+ + "[ -e \"$memory/online\" ] || continue; echo online > \"$memory/online\"; done'";
+
+ @Override
+ public boolean requiresKubeletRefresh(ServiceOffering oldOffering, ServiceOffering newOffering,
+ KubernetesClusterNodeType nodeType, KubernetesCluster.State originalState) {
+ return KubernetesCluster.State.Running == originalState
+ && (KubernetesClusterNodeType.WORKER == nodeType || KubernetesClusterNodeType.CONTROL == nodeType)
+ && capacityChanged(oldOffering, newOffering);
+ }
+
+ @Override
+ public NodeCapacitySnapshot captureBefore(KubernetesCluster cluster, UserVm vm, NodeAccess access) {
+ return nodeSnapshot(cluster, vm, access, guestResources(access));
+ }
+
+ @Override
+ public void cordonIfNeeded(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "=" + cluster.getUuid() + " --overwrite");
+ executeControl(access, "sudo /opt/bin/kubectl cordon " + node);
+ while (System.currentTimeMillis() < deadline) {
+ if (nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("CORDON", vm, "Kubernetes node did not become unschedulable");
+ }
+
+ @Override
+ public void onlineGuestResources(NodeAccess access) {
+ executeNode(access, ONLINE_HOTPLUGGED_RESOURCES);
+ }
+
+ @Override
+ public void verifyGuestResources(UserVm vm, ServiceOffering target, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ if (guestMatchesTarget(guestResources(access), target)) return;
+ sleep();
+ }
+ throw failure("GUEST_VERIFY", vm, "guest CPU or memory did not reach the target offering");
+ }
+
+ @Override
+ public boolean isKubernetesResourcesCurrent(NodeCapacitySnapshot snapshot, ServiceOffering target) {
+ GuestResources guest = new GuestResources(snapshot.getGuestOnlineCpuCount(), snapshot.getGuestMemoryKiB());
+ return guestMatchesTarget(guest, target) && kubernetesMatchesGuest(snapshot, snapshot);
+ }
+
+ @Override
+ public void restartKubelet(UserVm vm, NodeAccess access, long deadline) {
+ executeNode(access, "sudo systemctl restart kubelet");
+ while (System.currentTimeMillis() < deadline) {
+ if ("active".equals(executeNode(access, "sudo systemctl is-active kubelet").trim())) return;
+ sleep();
+ }
+ throw failure("KUBELET_RESTART", vm, "kubelet did not become active");
+ }
+
+ @Override
+ public NodeCapacitySnapshot waitForKubernetesResources(KubernetesCluster cluster, UserVm vm, ServiceOffering target,
+ NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ GuestResources guest = guestResources(access);
+ NodeCapacitySnapshot observed = nodeSnapshot(cluster, vm, access, guest);
+ if (guestMatchesTarget(guest, target) && kubernetesMatchesGuest(observed, before)) return observed;
+ sleep();
+ }
+ throw failure("CAPACITY_VERIFY", vm, "Kubernetes capacity did not match the resized guest");
+ }
+
+ @Override
+ public void restoreSchedulability(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl uncordon " + node);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "-");
+ while (System.currentTimeMillis() < deadline) {
+ if (!nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("RESTORE_SCHEDULABILITY", vm, "Kubernetes node remained unschedulable");
+ }
+
+ public static boolean capacityChanged(ServiceOffering oldOffering, ServiceOffering targetOffering) {
+ return oldOffering != null && targetOffering != null
+ && (!Objects.equals(oldOffering.getCpu(), targetOffering.getCpu())
+ || !Objects.equals(oldOffering.getRamSize(), targetOffering.getRamSize()));
+ }
+
+ static long parseCpuMillis(String value) { return value.endsWith("m") ? Long.parseLong(value.substring(0, value.length() - 1)) : Long.parseLong(value) * 1000L; }
+ static long parseMemoryBytes(String value) {
+ Matcher matcher = MEMORY_QUANTITY.matcher(value);
+ if (!matcher.matches()) throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ long number = Long.parseLong(matcher.group(1)); String unit = matcher.group(2);
+ if (unit == null) return number;
+ switch (unit) {
+ case "Ki": return number * 1024L; case "Mi": return number * 1024L * 1024L; case "Gi": return number * 1024L * 1024L * 1024L;
+ case "Ti": return number * 1024L * 1024L * 1024L * 1024L; case "K": return number * 1000L; case "M": return number * 1000L * 1000L;
+ case "G": return number * 1000L * 1000L * 1000L; case "T": return number * 1000L * 1000L * 1000L * 1000L;
+ default: throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
Review Comment:
Done, one case per line, using the `ByteScaleUtils` constants for the binary units (TiB is derived from them, since there's no constant for it). The decimal units K, M, G, T stay as powers of 1000.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "andrijapanicsb (via GitHub)" <gi...@apache.org>.
andrijapanicsb commented on code in PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#discussion_r4381572640
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterScaleWorker.java:
##########
@@ -361,7 +361,7 @@ private void validateKubernetesClusterScaleSizeParameters(KubernetesClusterNodeT
}
}
- private void scaleKubernetesClusterOffering(KubernetesClusterNodeType nodeType, ServiceOffering serviceOffering,
+ protected void scaleKubernetesClusterOffering(KubernetesClusterNodeType nodeType, ServiceOffering serviceOffering,
Review Comment:
I dont see any sonar warnings, but let me address your concerns
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6431967359
<b>[SF] Trillian test result (tid-17260)</b>
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 60816 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr14223-t17260-kvm-ol8.zip
Smoke tests completed. 156 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
test_01_list_sec_storage_vm | `Failure` | 0.07 | test_ssvm.py
test_02_list_cpvm_vm | `Failure` | 0.05 | test_ssvm.py
test_03_ssvm_internals | `Failure` | 0.05 | test_ssvm.py
test_04_cpvm_internals | `Failure` | 0.05 | test_ssvm.py
test_05_stop_ssvm | `Failure` | 0.05 | test_ssvm.py
test_06_stop_cpvm | `Failure` | 0.05 | test_ssvm.py
test_07_reboot_ssvm | `Failure` | 0.06 | test_ssvm.py
test_08_reboot_cpvm | `Failure` | 0.06 | test_ssvm.py
test_09_reboot_ssvm_forced | `Failure` | 0.06 | test_ssvm.py
test_10_reboot_cpvm_forced | `Failure` | 0.05 | test_ssvm.py
test_11_destroy_ssvm | `Failure` | 0.05 | test_ssvm.py
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "DaanHoogland (via GitHub) via commits" <co...@cloudstack.apache.org>.
DaanHoogland commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6719809836
@blueorangutan test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6469964992
> that is a 1:1 binary compatible replacement for VMware's VDDK
@andrijapanicsb , looks like a compatitor of the one mentioned in https://www.networkworld.com/article/4241265/fluidcloud-replaces-vddk-with-its-own-disk-copy-engine-for-vmware-exits.html
needs a comparison? (in a discussion item, probably not here)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "andrijapanicsb (via GitHub)" <gi...@apache.org>.
andrijapanicsb commented on code in PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#discussion_r4394976012
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/utils/KubernetesClusterNodeCapacityReconcilerImpl.java:
##########
@@ -0,0 +1,232 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.kubernetes.cluster.utils;
+
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import com.cloud.kubernetes.cluster.KubernetesCluster;
+import com.cloud.kubernetes.cluster.KubernetesServiceHelper.KubernetesClusterNodeType;
+import com.cloud.offering.ServiceOffering;
+import com.cloud.uservm.UserVm;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.ssh.SshHelper;
+import com.google.gson.JsonArray;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParser;
+import org.apache.commons.lang3.StringUtils;
+
+/** SSH-backed implementation for a rolling CKS live-resize reconciliation. */
+public class KubernetesClusterNodeCapacityReconcilerImpl implements KubernetesClusterNodeCapacityReconciler {
+ // KVM QA observes 4,568,788 KiB in a 4,608 MiB guest after memory hotplug
+ // (about 146 MiB below the offering), so keep a small margin above that
+ // platform overhead while still rejecting materially undersized guests.
+ static final long MINIMUM_MEMORY_OVERHEAD_BYTES = 160L * 1024L * 1024L;
+ static final long KUBERNETES_MEMORY_REPORTING_TOLERANCE_BYTES = 16L * 1024L * 1024L;
+ private static final int COMMAND_TIMEOUT_MS = 30000;
+ private static final int POLL_INTERVAL_MS = 5000;
+ private static final Pattern NODE_NAME = Pattern.compile("[A-Za-z0-9][A-Za-z0-9.-]{0,252}");
+ private static final Pattern MEMORY_QUANTITY = Pattern.compile("([0-9]+)(Ki|Mi|Gi|Ti|K|M|G|T)?");
+ private static final String RESIZE_ANNOTATION = "cloudstack.apache.org/cks-live-resize";
+ static final String ONLINE_HOTPLUGGED_RESOURCES = "sudo sh -c 'for cpu in /sys/devices/system/cpu/cpu[0-9]*; do "
+ + "[ -e \"$cpu/online\" ] || continue; echo 1 > \"$cpu/online\"; done; "
+ + "for memory in /sys/devices/system/memory/memory[0-9]*; do "
+ + "[ -e \"$memory/online\" ] || continue; echo online > \"$memory/online\"; done'";
+
+ @Override
+ public boolean requiresKubeletRefresh(ServiceOffering oldOffering, ServiceOffering newOffering,
+ KubernetesClusterNodeType nodeType, KubernetesCluster.State originalState) {
+ return KubernetesCluster.State.Running == originalState
+ && (KubernetesClusterNodeType.WORKER == nodeType || KubernetesClusterNodeType.CONTROL == nodeType)
+ && capacityChanged(oldOffering, newOffering);
+ }
+
+ @Override
+ public NodeCapacitySnapshot captureBefore(KubernetesCluster cluster, UserVm vm, NodeAccess access) {
+ return nodeSnapshot(cluster, vm, access, guestResources(access));
+ }
+
+ @Override
+ public void cordonIfNeeded(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "=" + cluster.getUuid() + " --overwrite");
+ executeControl(access, "sudo /opt/bin/kubectl cordon " + node);
+ while (System.currentTimeMillis() < deadline) {
+ if (nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("CORDON", vm, "Kubernetes node did not become unschedulable");
+ }
+
+ @Override
+ public void onlineGuestResources(NodeAccess access) {
+ executeNode(access, ONLINE_HOTPLUGGED_RESOURCES);
+ }
+
+ @Override
+ public void verifyGuestResources(UserVm vm, ServiceOffering target, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ if (guestMatchesTarget(guestResources(access), target)) return;
+ sleep();
+ }
+ throw failure("GUEST_VERIFY", vm, "guest CPU or memory did not reach the target offering");
+ }
+
+ @Override
+ public boolean isKubernetesResourcesCurrent(NodeCapacitySnapshot snapshot, ServiceOffering target) {
+ GuestResources guest = new GuestResources(snapshot.getGuestOnlineCpuCount(), snapshot.getGuestMemoryKiB());
+ return guestMatchesTarget(guest, target) && kubernetesMatchesGuest(snapshot, snapshot);
+ }
+
+ @Override
+ public void restartKubelet(UserVm vm, NodeAccess access, long deadline) {
+ executeNode(access, "sudo systemctl restart kubelet");
+ while (System.currentTimeMillis() < deadline) {
+ if ("active".equals(executeNode(access, "sudo systemctl is-active kubelet").trim())) return;
+ sleep();
+ }
+ throw failure("KUBELET_RESTART", vm, "kubelet did not become active");
+ }
+
+ @Override
+ public NodeCapacitySnapshot waitForKubernetesResources(KubernetesCluster cluster, UserVm vm, ServiceOffering target,
+ NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ GuestResources guest = guestResources(access);
+ NodeCapacitySnapshot observed = nodeSnapshot(cluster, vm, access, guest);
+ if (guestMatchesTarget(guest, target) && kubernetesMatchesGuest(observed, before)) return observed;
+ sleep();
+ }
+ throw failure("CAPACITY_VERIFY", vm, "Kubernetes capacity did not match the resized guest");
+ }
+
+ @Override
+ public void restoreSchedulability(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl uncordon " + node);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "-");
+ while (System.currentTimeMillis() < deadline) {
+ if (!nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("RESTORE_SCHEDULABILITY", vm, "Kubernetes node remained unschedulable");
+ }
+
+ public static boolean capacityChanged(ServiceOffering oldOffering, ServiceOffering targetOffering) {
+ return oldOffering != null && targetOffering != null
+ && (!Objects.equals(oldOffering.getCpu(), targetOffering.getCpu())
+ || !Objects.equals(oldOffering.getRamSize(), targetOffering.getRamSize()));
+ }
+
+ static long parseCpuMillis(String value) { return value.endsWith("m") ? Long.parseLong(value.substring(0, value.length() - 1)) : Long.parseLong(value) * 1000L; }
+ static long parseMemoryBytes(String value) {
+ Matcher matcher = MEMORY_QUANTITY.matcher(value);
+ if (!matcher.matches()) throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ long number = Long.parseLong(matcher.group(1)); String unit = matcher.group(2);
+ if (unit == null) return number;
+ switch (unit) {
+ case "Ki": return number * 1024L; case "Mi": return number * 1024L * 1024L; case "Gi": return number * 1024L * 1024L * 1024L;
+ case "Ti": return number * 1024L * 1024L * 1024L * 1024L; case "K": return number * 1000L; case "M": return number * 1000L * 1000L;
+ case "G": return number * 1000L * 1000L * 1000L; case "T": return number * 1000L * 1000L * 1000L * 1000L;
+ default: throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ }
+ }
+
+ private NodeCapacitySnapshot nodeSnapshot(KubernetesCluster cluster, UserVm vm, NodeAccess access, GuestResources guest) {
+ JsonObject node = new JsonParser().parse(executeControl(access, "sudo /opt/bin/kubectl get node " + nodeName(vm) + " -o json")).getAsJsonObject();
Review Comment:
Done, it uses `JsonParser.parseString` now.
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/utils/KubernetesClusterNodeCapacityReconcilerImpl.java:
##########
@@ -0,0 +1,232 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.kubernetes.cluster.utils;
+
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import com.cloud.kubernetes.cluster.KubernetesCluster;
+import com.cloud.kubernetes.cluster.KubernetesServiceHelper.KubernetesClusterNodeType;
+import com.cloud.offering.ServiceOffering;
+import com.cloud.uservm.UserVm;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.ssh.SshHelper;
+import com.google.gson.JsonArray;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParser;
+import org.apache.commons.lang3.StringUtils;
+
+/** SSH-backed implementation for a rolling CKS live-resize reconciliation. */
+public class KubernetesClusterNodeCapacityReconcilerImpl implements KubernetesClusterNodeCapacityReconciler {
+ // KVM QA observes 4,568,788 KiB in a 4,608 MiB guest after memory hotplug
+ // (about 146 MiB below the offering), so keep a small margin above that
+ // platform overhead while still rejecting materially undersized guests.
+ static final long MINIMUM_MEMORY_OVERHEAD_BYTES = 160L * 1024L * 1024L;
+ static final long KUBERNETES_MEMORY_REPORTING_TOLERANCE_BYTES = 16L * 1024L * 1024L;
+ private static final int COMMAND_TIMEOUT_MS = 30000;
+ private static final int POLL_INTERVAL_MS = 5000;
+ private static final Pattern NODE_NAME = Pattern.compile("[A-Za-z0-9][A-Za-z0-9.-]{0,252}");
+ private static final Pattern MEMORY_QUANTITY = Pattern.compile("([0-9]+)(Ki|Mi|Gi|Ti|K|M|G|T)?");
+ private static final String RESIZE_ANNOTATION = "cloudstack.apache.org/cks-live-resize";
+ static final String ONLINE_HOTPLUGGED_RESOURCES = "sudo sh -c 'for cpu in /sys/devices/system/cpu/cpu[0-9]*; do "
+ + "[ -e \"$cpu/online\" ] || continue; echo 1 > \"$cpu/online\"; done; "
+ + "for memory in /sys/devices/system/memory/memory[0-9]*; do "
+ + "[ -e \"$memory/online\" ] || continue; echo online > \"$memory/online\"; done'";
+
+ @Override
+ public boolean requiresKubeletRefresh(ServiceOffering oldOffering, ServiceOffering newOffering,
+ KubernetesClusterNodeType nodeType, KubernetesCluster.State originalState) {
+ return KubernetesCluster.State.Running == originalState
+ && (KubernetesClusterNodeType.WORKER == nodeType || KubernetesClusterNodeType.CONTROL == nodeType)
+ && capacityChanged(oldOffering, newOffering);
+ }
+
+ @Override
+ public NodeCapacitySnapshot captureBefore(KubernetesCluster cluster, UserVm vm, NodeAccess access) {
+ return nodeSnapshot(cluster, vm, access, guestResources(access));
+ }
+
+ @Override
+ public void cordonIfNeeded(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "=" + cluster.getUuid() + " --overwrite");
+ executeControl(access, "sudo /opt/bin/kubectl cordon " + node);
+ while (System.currentTimeMillis() < deadline) {
+ if (nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("CORDON", vm, "Kubernetes node did not become unschedulable");
+ }
+
+ @Override
+ public void onlineGuestResources(NodeAccess access) {
+ executeNode(access, ONLINE_HOTPLUGGED_RESOURCES);
+ }
+
+ @Override
+ public void verifyGuestResources(UserVm vm, ServiceOffering target, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ if (guestMatchesTarget(guestResources(access), target)) return;
+ sleep();
+ }
+ throw failure("GUEST_VERIFY", vm, "guest CPU or memory did not reach the target offering");
+ }
+
+ @Override
+ public boolean isKubernetesResourcesCurrent(NodeCapacitySnapshot snapshot, ServiceOffering target) {
+ GuestResources guest = new GuestResources(snapshot.getGuestOnlineCpuCount(), snapshot.getGuestMemoryKiB());
+ return guestMatchesTarget(guest, target) && kubernetesMatchesGuest(snapshot, snapshot);
+ }
+
+ @Override
+ public void restartKubelet(UserVm vm, NodeAccess access, long deadline) {
+ executeNode(access, "sudo systemctl restart kubelet");
+ while (System.currentTimeMillis() < deadline) {
+ if ("active".equals(executeNode(access, "sudo systemctl is-active kubelet").trim())) return;
+ sleep();
+ }
+ throw failure("KUBELET_RESTART", vm, "kubelet did not become active");
+ }
+
+ @Override
+ public NodeCapacitySnapshot waitForKubernetesResources(KubernetesCluster cluster, UserVm vm, ServiceOffering target,
+ NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ GuestResources guest = guestResources(access);
+ NodeCapacitySnapshot observed = nodeSnapshot(cluster, vm, access, guest);
+ if (guestMatchesTarget(guest, target) && kubernetesMatchesGuest(observed, before)) return observed;
+ sleep();
+ }
+ throw failure("CAPACITY_VERIFY", vm, "Kubernetes capacity did not match the resized guest");
+ }
+
+ @Override
+ public void restoreSchedulability(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl uncordon " + node);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "-");
+ while (System.currentTimeMillis() < deadline) {
+ if (!nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("RESTORE_SCHEDULABILITY", vm, "Kubernetes node remained unschedulable");
+ }
+
+ public static boolean capacityChanged(ServiceOffering oldOffering, ServiceOffering targetOffering) {
+ return oldOffering != null && targetOffering != null
+ && (!Objects.equals(oldOffering.getCpu(), targetOffering.getCpu())
+ || !Objects.equals(oldOffering.getRamSize(), targetOffering.getRamSize()));
+ }
+
+ static long parseCpuMillis(String value) { return value.endsWith("m") ? Long.parseLong(value.substring(0, value.length() - 1)) : Long.parseLong(value) * 1000L; }
+ static long parseMemoryBytes(String value) {
+ Matcher matcher = MEMORY_QUANTITY.matcher(value);
+ if (!matcher.matches()) throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ long number = Long.parseLong(matcher.group(1)); String unit = matcher.group(2);
+ if (unit == null) return number;
+ switch (unit) {
+ case "Ki": return number * 1024L; case "Mi": return number * 1024L * 1024L; case "Gi": return number * 1024L * 1024L * 1024L;
+ case "Ti": return number * 1024L * 1024L * 1024L * 1024L; case "K": return number * 1000L; case "M": return number * 1000L * 1000L;
+ case "G": return number * 1000L * 1000L * 1000L; case "T": return number * 1000L * 1000L * 1000L * 1000L;
+ default: throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ }
+ }
+
+ private NodeCapacitySnapshot nodeSnapshot(KubernetesCluster cluster, UserVm vm, NodeAccess access, GuestResources guest) {
+ JsonObject node = new JsonParser().parse(executeControl(access, "sudo /opt/bin/kubectl get node " + nodeName(vm) + " -o json")).getAsJsonObject();
+ JsonObject annotations = node.getAsJsonObject("metadata").has("annotations") ? node.getAsJsonObject("metadata").getAsJsonObject("annotations") : null;
+ boolean cksCordon = annotations != null && annotations.has(RESIZE_ANNOTATION) && cluster.getUuid().equals(annotations.get(RESIZE_ANNOTATION).getAsString());
+ JsonObject spec = node.has("spec") ? node.getAsJsonObject("spec") : new JsonObject(); JsonObject status = node.getAsJsonObject("status");
Review Comment:
Done.
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/utils/KubernetesClusterNodeCapacityReconcilerImpl.java:
##########
@@ -0,0 +1,232 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.kubernetes.cluster.utils;
+
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import com.cloud.kubernetes.cluster.KubernetesCluster;
+import com.cloud.kubernetes.cluster.KubernetesServiceHelper.KubernetesClusterNodeType;
+import com.cloud.offering.ServiceOffering;
+import com.cloud.uservm.UserVm;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.ssh.SshHelper;
+import com.google.gson.JsonArray;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParser;
+import org.apache.commons.lang3.StringUtils;
+
+/** SSH-backed implementation for a rolling CKS live-resize reconciliation. */
+public class KubernetesClusterNodeCapacityReconcilerImpl implements KubernetesClusterNodeCapacityReconciler {
+ // KVM QA observes 4,568,788 KiB in a 4,608 MiB guest after memory hotplug
+ // (about 146 MiB below the offering), so keep a small margin above that
+ // platform overhead while still rejecting materially undersized guests.
+ static final long MINIMUM_MEMORY_OVERHEAD_BYTES = 160L * 1024L * 1024L;
+ static final long KUBERNETES_MEMORY_REPORTING_TOLERANCE_BYTES = 16L * 1024L * 1024L;
+ private static final int COMMAND_TIMEOUT_MS = 30000;
+ private static final int POLL_INTERVAL_MS = 5000;
+ private static final Pattern NODE_NAME = Pattern.compile("[A-Za-z0-9][A-Za-z0-9.-]{0,252}");
+ private static final Pattern MEMORY_QUANTITY = Pattern.compile("([0-9]+)(Ki|Mi|Gi|Ti|K|M|G|T)?");
+ private static final String RESIZE_ANNOTATION = "cloudstack.apache.org/cks-live-resize";
+ static final String ONLINE_HOTPLUGGED_RESOURCES = "sudo sh -c 'for cpu in /sys/devices/system/cpu/cpu[0-9]*; do "
+ + "[ -e \"$cpu/online\" ] || continue; echo 1 > \"$cpu/online\"; done; "
+ + "for memory in /sys/devices/system/memory/memory[0-9]*; do "
+ + "[ -e \"$memory/online\" ] || continue; echo online > \"$memory/online\"; done'";
+
+ @Override
+ public boolean requiresKubeletRefresh(ServiceOffering oldOffering, ServiceOffering newOffering,
+ KubernetesClusterNodeType nodeType, KubernetesCluster.State originalState) {
+ return KubernetesCluster.State.Running == originalState
+ && (KubernetesClusterNodeType.WORKER == nodeType || KubernetesClusterNodeType.CONTROL == nodeType)
+ && capacityChanged(oldOffering, newOffering);
+ }
+
+ @Override
+ public NodeCapacitySnapshot captureBefore(KubernetesCluster cluster, UserVm vm, NodeAccess access) {
+ return nodeSnapshot(cluster, vm, access, guestResources(access));
+ }
+
+ @Override
+ public void cordonIfNeeded(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "=" + cluster.getUuid() + " --overwrite");
+ executeControl(access, "sudo /opt/bin/kubectl cordon " + node);
+ while (System.currentTimeMillis() < deadline) {
+ if (nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("CORDON", vm, "Kubernetes node did not become unschedulable");
+ }
+
+ @Override
+ public void onlineGuestResources(NodeAccess access) {
+ executeNode(access, ONLINE_HOTPLUGGED_RESOURCES);
+ }
+
+ @Override
+ public void verifyGuestResources(UserVm vm, ServiceOffering target, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ if (guestMatchesTarget(guestResources(access), target)) return;
+ sleep();
+ }
+ throw failure("GUEST_VERIFY", vm, "guest CPU or memory did not reach the target offering");
+ }
+
+ @Override
+ public boolean isKubernetesResourcesCurrent(NodeCapacitySnapshot snapshot, ServiceOffering target) {
+ GuestResources guest = new GuestResources(snapshot.getGuestOnlineCpuCount(), snapshot.getGuestMemoryKiB());
+ return guestMatchesTarget(guest, target) && kubernetesMatchesGuest(snapshot, snapshot);
+ }
+
+ @Override
+ public void restartKubelet(UserVm vm, NodeAccess access, long deadline) {
+ executeNode(access, "sudo systemctl restart kubelet");
+ while (System.currentTimeMillis() < deadline) {
+ if ("active".equals(executeNode(access, "sudo systemctl is-active kubelet").trim())) return;
+ sleep();
+ }
+ throw failure("KUBELET_RESTART", vm, "kubelet did not become active");
+ }
+
+ @Override
+ public NodeCapacitySnapshot waitForKubernetesResources(KubernetesCluster cluster, UserVm vm, ServiceOffering target,
+ NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ GuestResources guest = guestResources(access);
+ NodeCapacitySnapshot observed = nodeSnapshot(cluster, vm, access, guest);
+ if (guestMatchesTarget(guest, target) && kubernetesMatchesGuest(observed, before)) return observed;
+ sleep();
+ }
+ throw failure("CAPACITY_VERIFY", vm, "Kubernetes capacity did not match the resized guest");
+ }
+
+ @Override
+ public void restoreSchedulability(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl uncordon " + node);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "-");
+ while (System.currentTimeMillis() < deadline) {
+ if (!nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("RESTORE_SCHEDULABILITY", vm, "Kubernetes node remained unschedulable");
+ }
+
+ public static boolean capacityChanged(ServiceOffering oldOffering, ServiceOffering targetOffering) {
+ return oldOffering != null && targetOffering != null
+ && (!Objects.equals(oldOffering.getCpu(), targetOffering.getCpu())
+ || !Objects.equals(oldOffering.getRamSize(), targetOffering.getRamSize()));
+ }
+
+ static long parseCpuMillis(String value) { return value.endsWith("m") ? Long.parseLong(value.substring(0, value.length() - 1)) : Long.parseLong(value) * 1000L; }
+ static long parseMemoryBytes(String value) {
+ Matcher matcher = MEMORY_QUANTITY.matcher(value);
+ if (!matcher.matches()) throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ long number = Long.parseLong(matcher.group(1)); String unit = matcher.group(2);
+ if (unit == null) return number;
+ switch (unit) {
+ case "Ki": return number * 1024L; case "Mi": return number * 1024L * 1024L; case "Gi": return number * 1024L * 1024L * 1024L;
+ case "Ti": return number * 1024L * 1024L * 1024L * 1024L; case "K": return number * 1000L; case "M": return number * 1000L * 1000L;
+ case "G": return number * 1000L * 1000L * 1000L; case "T": return number * 1000L * 1000L * 1000L * 1000L;
+ default: throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ }
+ }
+
+ private NodeCapacitySnapshot nodeSnapshot(KubernetesCluster cluster, UserVm vm, NodeAccess access, GuestResources guest) {
+ JsonObject node = new JsonParser().parse(executeControl(access, "sudo /opt/bin/kubectl get node " + nodeName(vm) + " -o json")).getAsJsonObject();
+ JsonObject annotations = node.getAsJsonObject("metadata").has("annotations") ? node.getAsJsonObject("metadata").getAsJsonObject("annotations") : null;
+ boolean cksCordon = annotations != null && annotations.has(RESIZE_ANNOTATION) && cluster.getUuid().equals(annotations.get(RESIZE_ANNOTATION).getAsString());
+ JsonObject spec = node.has("spec") ? node.getAsJsonObject("spec") : new JsonObject(); JsonObject status = node.getAsJsonObject("status");
+ JsonObject capacity = status.getAsJsonObject("capacity"); JsonObject allocatable = status.getAsJsonObject("allocatable");
Review Comment:
Done.
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/utils/KubernetesClusterNodeCapacityReconcilerImpl.java:
##########
@@ -0,0 +1,232 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.kubernetes.cluster.utils;
+
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import com.cloud.kubernetes.cluster.KubernetesCluster;
+import com.cloud.kubernetes.cluster.KubernetesServiceHelper.KubernetesClusterNodeType;
+import com.cloud.offering.ServiceOffering;
+import com.cloud.uservm.UserVm;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.ssh.SshHelper;
+import com.google.gson.JsonArray;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParser;
+import org.apache.commons.lang3.StringUtils;
+
+/** SSH-backed implementation for a rolling CKS live-resize reconciliation. */
+public class KubernetesClusterNodeCapacityReconcilerImpl implements KubernetesClusterNodeCapacityReconciler {
+ // KVM QA observes 4,568,788 KiB in a 4,608 MiB guest after memory hotplug
+ // (about 146 MiB below the offering), so keep a small margin above that
+ // platform overhead while still rejecting materially undersized guests.
+ static final long MINIMUM_MEMORY_OVERHEAD_BYTES = 160L * 1024L * 1024L;
+ static final long KUBERNETES_MEMORY_REPORTING_TOLERANCE_BYTES = 16L * 1024L * 1024L;
+ private static final int COMMAND_TIMEOUT_MS = 30000;
+ private static final int POLL_INTERVAL_MS = 5000;
+ private static final Pattern NODE_NAME = Pattern.compile("[A-Za-z0-9][A-Za-z0-9.-]{0,252}");
+ private static final Pattern MEMORY_QUANTITY = Pattern.compile("([0-9]+)(Ki|Mi|Gi|Ti|K|M|G|T)?");
+ private static final String RESIZE_ANNOTATION = "cloudstack.apache.org/cks-live-resize";
+ static final String ONLINE_HOTPLUGGED_RESOURCES = "sudo sh -c 'for cpu in /sys/devices/system/cpu/cpu[0-9]*; do "
+ + "[ -e \"$cpu/online\" ] || continue; echo 1 > \"$cpu/online\"; done; "
+ + "for memory in /sys/devices/system/memory/memory[0-9]*; do "
+ + "[ -e \"$memory/online\" ] || continue; echo online > \"$memory/online\"; done'";
+
+ @Override
+ public boolean requiresKubeletRefresh(ServiceOffering oldOffering, ServiceOffering newOffering,
+ KubernetesClusterNodeType nodeType, KubernetesCluster.State originalState) {
+ return KubernetesCluster.State.Running == originalState
+ && (KubernetesClusterNodeType.WORKER == nodeType || KubernetesClusterNodeType.CONTROL == nodeType)
+ && capacityChanged(oldOffering, newOffering);
+ }
+
+ @Override
+ public NodeCapacitySnapshot captureBefore(KubernetesCluster cluster, UserVm vm, NodeAccess access) {
+ return nodeSnapshot(cluster, vm, access, guestResources(access));
+ }
+
+ @Override
+ public void cordonIfNeeded(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "=" + cluster.getUuid() + " --overwrite");
+ executeControl(access, "sudo /opt/bin/kubectl cordon " + node);
+ while (System.currentTimeMillis() < deadline) {
+ if (nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("CORDON", vm, "Kubernetes node did not become unschedulable");
+ }
+
+ @Override
+ public void onlineGuestResources(NodeAccess access) {
+ executeNode(access, ONLINE_HOTPLUGGED_RESOURCES);
+ }
+
+ @Override
+ public void verifyGuestResources(UserVm vm, ServiceOffering target, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ if (guestMatchesTarget(guestResources(access), target)) return;
+ sleep();
+ }
+ throw failure("GUEST_VERIFY", vm, "guest CPU or memory did not reach the target offering");
+ }
+
+ @Override
+ public boolean isKubernetesResourcesCurrent(NodeCapacitySnapshot snapshot, ServiceOffering target) {
+ GuestResources guest = new GuestResources(snapshot.getGuestOnlineCpuCount(), snapshot.getGuestMemoryKiB());
+ return guestMatchesTarget(guest, target) && kubernetesMatchesGuest(snapshot, snapshot);
+ }
+
+ @Override
+ public void restartKubelet(UserVm vm, NodeAccess access, long deadline) {
+ executeNode(access, "sudo systemctl restart kubelet");
+ while (System.currentTimeMillis() < deadline) {
+ if ("active".equals(executeNode(access, "sudo systemctl is-active kubelet").trim())) return;
+ sleep();
+ }
+ throw failure("KUBELET_RESTART", vm, "kubelet did not become active");
+ }
+
+ @Override
+ public NodeCapacitySnapshot waitForKubernetesResources(KubernetesCluster cluster, UserVm vm, ServiceOffering target,
+ NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ GuestResources guest = guestResources(access);
+ NodeCapacitySnapshot observed = nodeSnapshot(cluster, vm, access, guest);
+ if (guestMatchesTarget(guest, target) && kubernetesMatchesGuest(observed, before)) return observed;
+ sleep();
+ }
+ throw failure("CAPACITY_VERIFY", vm, "Kubernetes capacity did not match the resized guest");
+ }
+
+ @Override
+ public void restoreSchedulability(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl uncordon " + node);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "-");
+ while (System.currentTimeMillis() < deadline) {
+ if (!nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("RESTORE_SCHEDULABILITY", vm, "Kubernetes node remained unschedulable");
+ }
+
+ public static boolean capacityChanged(ServiceOffering oldOffering, ServiceOffering targetOffering) {
+ return oldOffering != null && targetOffering != null
+ && (!Objects.equals(oldOffering.getCpu(), targetOffering.getCpu())
+ || !Objects.equals(oldOffering.getRamSize(), targetOffering.getRamSize()));
+ }
+
+ static long parseCpuMillis(String value) { return value.endsWith("m") ? Long.parseLong(value.substring(0, value.length() - 1)) : Long.parseLong(value) * 1000L; }
+ static long parseMemoryBytes(String value) {
+ Matcher matcher = MEMORY_QUANTITY.matcher(value);
+ if (!matcher.matches()) throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ long number = Long.parseLong(matcher.group(1)); String unit = matcher.group(2);
+ if (unit == null) return number;
+ switch (unit) {
+ case "Ki": return number * 1024L; case "Mi": return number * 1024L * 1024L; case "Gi": return number * 1024L * 1024L * 1024L;
+ case "Ti": return number * 1024L * 1024L * 1024L * 1024L; case "K": return number * 1000L; case "M": return number * 1000L * 1000L;
+ case "G": return number * 1000L * 1000L * 1000L; case "T": return number * 1000L * 1000L * 1000L * 1000L;
+ default: throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ }
+ }
+
+ private NodeCapacitySnapshot nodeSnapshot(KubernetesCluster cluster, UserVm vm, NodeAccess access, GuestResources guest) {
+ JsonObject node = new JsonParser().parse(executeControl(access, "sudo /opt/bin/kubectl get node " + nodeName(vm) + " -o json")).getAsJsonObject();
+ JsonObject annotations = node.getAsJsonObject("metadata").has("annotations") ? node.getAsJsonObject("metadata").getAsJsonObject("annotations") : null;
+ boolean cksCordon = annotations != null && annotations.has(RESIZE_ANNOTATION) && cluster.getUuid().equals(annotations.get(RESIZE_ANNOTATION).getAsString());
+ JsonObject spec = node.has("spec") ? node.getAsJsonObject("spec") : new JsonObject(); JsonObject status = node.getAsJsonObject("status");
+ JsonObject capacity = status.getAsJsonObject("capacity"); JsonObject allocatable = status.getAsJsonObject("allocatable");
+ return new NodeCapacitySnapshot(spec.has("unschedulable") && spec.get("unschedulable").getAsBoolean(), cksCordon,
+ guest == null ? 0 : guest.cpu, guest == null ? 0 : guest.memoryKiB, parseCpuMillis(capacity.get("cpu").getAsString()),
+ parseMemoryBytes(capacity.get("memory").getAsString()), parseCpuMillis(allocatable.get("cpu").getAsString()),
+ parseMemoryBytes(allocatable.get("memory").getAsString()), isReady(status));
Review Comment:
Done, one argument per line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "andrijapanicsb (via GitHub)" <gi...@apache.org>.
andrijapanicsb commented on code in PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#discussion_r4394973071
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterScaleWorker.java:
##########
@@ -376,24 +377,171 @@ private void scaleKubernetesClusterOffering(KubernetesClusterNodeType nodeType,
List<KubernetesClusterVmMapVO> vmList = kubernetesClusterVmMapDao.listByClusterIdAndVmType(kubernetesCluster.getId(), nodeType);
final long tobeScaledVMCount = Math.min(vmList.size(), size);
for (long i = 0; i < tobeScaledVMCount; i++) {
- KubernetesClusterVmMapVO vmMapVO = vmList.get((int) i);
- UserVmVO userVM = userVmDao.findById(vmMapVO.getVmId());
- boolean result = false;
- try {
- result = userVmManager.upgradeVirtualMachine(userVM.getId(), serviceOffering.getId(), new HashMap<String, String>());
- } catch (RuntimeException | ResourceUnavailableException | ManagementServerException | VirtualMachineMigrationException e) {
- logTransitStateAndThrow(Level.ERROR, String.format("Scaling Kubernetes cluster : %s failed, unable to scale cluster VM : %s due to %s", kubernetesCluster.getName(), userVM.getDisplayName(), e.getMessage()), kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed, e);
- }
- if (!result) {
- logTransitStateAndThrow(Level.WARN, String.format("Scaling Kubernetes cluster : %s failed, unable to scale cluster VM : %s", kubernetesCluster.getName(), userVM.getDisplayName()),kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
- }
+ new NodeOfferingResize(vmList.get((int) i), nodeType, serviceOffering).execute();
if (System.currentTimeMillis() > scaleTimeoutTime) {
logTransitStateAndThrow(Level.WARN, String.format("Scaling Kubernetes cluster : %s failed, scaling action timed out", kubernetesCluster.getName()),kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
}
}
kubernetesCluster = updateKubernetesClusterEntryForNodeType(null, nodeType, serviceOffering, updateNodeOffering, updateClusterOffering);
}
+ private final class NodeOfferingResize {
+ private final KubernetesClusterVmMapVO vmMap;
+ private final KubernetesClusterNodeType nodeType;
+ private final ServiceOffering targetOffering;
+ private final ServiceOffering oldOffering;
+ private UserVmVO vm;
+ private NodeAccess access;
+ private NodeCapacitySnapshot before;
+ private boolean reconcileKubernetes;
+ private boolean cksCordonCompleted;
+ private boolean resizeSucceeded;
+
+ private NodeOfferingResize(KubernetesClusterVmMapVO vmMap, KubernetesClusterNodeType nodeType, ServiceOffering targetOffering) {
+ this.vmMap = vmMap;
+ this.nodeType = nodeType;
+ this.targetOffering = targetOffering;
+ vm = userVmDao.findById(vmMap.getVmId());
+ if (vm == null) {
+ logTransitStateAndThrow(Level.ERROR, String.format("Scaling Kubernetes cluster : %s failed, unable to find cluster VM %s",
+ kubernetesCluster.getName(), vmMap.getVmId()), kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
+ }
+ oldOffering = serviceOfferingDao.findById(vm.getServiceOfferingId());
+ resizeSucceeded = Objects.equals(targetOffering.getId(), vm.getServiceOfferingId());
+ }
+
+ private void execute() {
+ validateLiveResize();
+ try {
+ prepareReconciliation();
+ resizeVm();
+ reconcileCapacity();
+ } catch (ResourceUnavailableException | ManagementServerException | VirtualMachineMigrationException | CloudRuntimeException e) {
+ restoreAfterFailedResize();
+ reportFailure(e);
+ }
+ }
+
+ private void validateLiveResize() {
+ if (KubernetesClusterNodeCapacityReconcilerImpl.capacityChanged(oldOffering, targetOffering)
+ && KubernetesCluster.State.Running == originalState && ETCD == nodeType
+ && KubernetesCluster.ClusterType.CloudManaged == kubernetesCluster.getClusterType()) {
+ logTransitStateAndThrow(Level.ERROR, String.format("Scaling Kubernetes cluster : %s cannot live-resize dedicated etcd VM %s; " +
+ "etcd health reconciliation is not implemented", kubernetesCluster.getName(), vm.getDisplayName()),
+ kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
+ }
+ }
+
+ private void prepareReconciliation() {
+ reconcileKubernetes = shouldReconcileNodeCapacity(vmMap, vm, oldOffering, targetOffering, nodeType);
+ if (reconcileKubernetes || isReconciliationRetryCandidate(vmMap, vm, targetOffering, nodeType)) {
+ access = resolveNodeAccess(vm);
+ before = kubernetesClusterNodeCapacityReconciler.captureBefore(kubernetesCluster, vm, access);
+ reconcileKubernetes = reconcileKubernetes || before.isCloudStackResizeCordon();
+ }
+ if (reconcileKubernetes) {
+ kubernetesClusterNodeCapacityReconciler.cordonIfNeeded(kubernetesCluster, vm, before, access, scaleTimeoutTime);
+ cksCordonCompleted = !before.isUnschedulable() || before.isCloudStackResizeCordon();
+ }
+ }
+
+ private void resizeVm() throws ResourceUnavailableException, ManagementServerException, VirtualMachineMigrationException {
+ if (resizeSucceeded) {
+ return;
+ }
+ if (!userVmManager.upgradeVirtualMachine(vm.getId(), targetOffering.getId(), new HashMap<>())) {
+ throw new CloudRuntimeException("Unable to scale cluster VM " + vm.getDisplayName());
+ }
+ resizeSucceeded = true;
+ UserVmVO resizedVm = userVmDao.findById(vm.getId());
+ if (resizedVm == null || !Objects.equals(targetOffering.getId(), resizedVm.getServiceOfferingId())) {
+ throw new CloudRuntimeException("VM " + vm.getUuid() + " did not reach target offering");
+ }
+ vm = resizedVm;
+ }
+
+ private void reconcileCapacity() {
+ if (!reconcileKubernetes) {
+ return;
+ }
+ kubernetesClusterNodeCapacityReconciler.onlineGuestResources(access);
+ kubernetesClusterNodeCapacityReconciler.verifyGuestResources(vm, targetOffering, before, access, scaleTimeoutTime);
+ if (!kubernetesClusterNodeCapacityReconciler.isKubernetesResourcesCurrent(before, targetOffering)) {
+ kubernetesClusterNodeCapacityReconciler.restartKubelet(vm, access, scaleTimeoutTime);
+ kubernetesClusterNodeCapacityReconciler.waitForKubernetesResources(kubernetesCluster, vm, targetOffering, before, access, scaleTimeoutTime);
+ }
+ kubernetesClusterNodeCapacityReconciler.restoreSchedulability(kubernetesCluster, vm, before, access, scaleTimeoutTime);
+ }
+
+ private void restoreAfterFailedResize() {
+ if (!reconcileKubernetes || !cksCordonCompleted || resizeSucceeded) {
+ return;
+ }
+ try {
+ kubernetesClusterNodeCapacityReconciler.restoreSchedulability(kubernetesCluster, vm, before, access, scaleTimeoutTime);
+ } catch (CloudRuntimeException cleanupException) {
+ logger.warn("Unable to restore schedulability for CKS node {} after a pre-resize failure", vm.getUuid(), cleanupException);
+ }
+ }
+
+ private void reportFailure(Exception cause) {
+ String recovery = resizeSucceeded
+ ? "The node remains cordoned; check kubelet on the VM and retry the CKS scale operation."
+ : "The VM was not resized; CKS attempted to restore its temporary cordon.";
+ String message = String.format("Scaling Kubernetes cluster %s (UUID %s) failed for %s node VM %s (UUID %s), " +
+ "offering %s to %s: %s %s", kubernetesCluster.getName(), kubernetesCluster.getUuid(), nodeType,
+ vm.getDisplayName(), vm.getUuid(), oldOffering == null ? null : oldOffering.getId(),
+ targetOffering.getId(), cause.getMessage(), recovery);
+ logTransitStateAndThrow(Level.ERROR, message, kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed, cause);
+ }
+ }
+
+ private NodeAccess resolveNodeAccess(UserVm userVM) {
+ Pair<String, Integer> controlAccess = getKubernetesClusterServerIpSshPort(null);
+ if (StringUtils.isBlank(controlAccess.first())) {
+ throw new CloudRuntimeException(String.format("Unable to resolve control-plane SSH for Kubernetes cluster %s", kubernetesCluster.getUuid()));
+ }
+ if (manager.isDirectAccess(network)) {
+ if (StringUtils.isBlank(userVM.getPrivateIpAddress())) {
+ throw new CloudRuntimeException(String.format("Unable to resolve private SSH address for VM %s", userVM.getUuid()));
+ }
+ return new NodeAccess(controlAccess.first(), controlAccess.second(), userVM.getPrivateIpAddress(), DEFAULT_SSH_PORT,
+ getControlNodeLoginUser(), getManagementServerSshPublicKeyFile());
+ }
+ PortForwardingRuleVO sshRule = portForwardingRulesDao.listByVm(userVM.getId()).stream()
+ .filter(rule -> rule.getDestinationPortStart() == DEFAULT_SSH_PORT)
+ .filter(rule -> !FirewallRule.State.Revoke.equals(rule.getState()))
+ .findFirst().orElse(null);
+ if (sshRule == null) {
+ throw new CloudRuntimeException(String.format("Unable to resolve SSH port-forwarding rule for VM %s", userVM.getUuid()));
+ }
+ IpAddress targetPublicIp = network.getVpcId() == null ? getNetworkSourceNatIp(network) : getVpcTierKubernetesPublicIp(network);
+ if (targetPublicIp == null) {
+ throw new CloudRuntimeException(String.format("Unable to resolve target-node public IP for VM %s", userVM.getUuid()));
+ }
+ return new NodeAccess(controlAccess.first(), controlAccess.second(), targetPublicIp.getAddress().addr(), sshRule.getSourcePortStart(),
+ getControlNodeLoginUser(), getManagementServerSshPublicKeyFile());
+ }
+
+ protected boolean shouldReconcileNodeCapacity(KubernetesClusterVmMapVO vmMapVO, UserVm userVM,
+ ServiceOffering oldOffering, ServiceOffering targetOffering,
+ KubernetesClusterNodeType nodeType) {
+ return !vmMapVO.isExternalNode()
+ && KubernetesCluster.ClusterType.CloudManaged == kubernetesCluster.getClusterType()
+ && Hypervisor.HypervisorType.KVM == userVM.getHypervisorType()
+ && kubernetesClusterNodeCapacityReconciler.requiresKubeletRefresh(oldOffering, targetOffering, nodeType, originalState);
+ }
+
+ protected boolean isReconciliationRetryCandidate(KubernetesClusterVmMapVO vmMapVO, UserVm userVM,
+ ServiceOffering targetOffering, KubernetesClusterNodeType nodeType) {
+ return !vmMapVO.isExternalNode()
+ && KubernetesCluster.ClusterType.CloudManaged == kubernetesCluster.getClusterType()
+ && Hypervisor.HypervisorType.KVM == userVM.getHypervisorType()
+ && (WORKER == nodeType || CONTROL == nodeType)
+ && KubernetesCluster.State.Running == originalState
Review Comment:
Done, the shared checks are in one method now, which also gives the reason for the log line. The already-on-target case only adds the check for the pending annotation.
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterScaleWorker.java:
##########
@@ -605,6 +753,9 @@ public boolean scaleCluster() throws CloudRuntimeException {
for (KubernetesClusterNodeType nodeType : Arrays.asList(CONTROL, ETCD, WORKER)) {
boolean isWorkerNode = WORKER == nodeType;
final long newVMRequired = (!isWorkerNode || clusterSize == null) ? 0 : clusterSize - originalClusterSize;
+ if (nodeType == ETCD && kubernetesCluster.getEtcdNodeCount() == 0) {
+ continue;
+ }
Review Comment:
Agreed, it belongs in #14138. Since #14138 is still a draft against 4.22, I kept the guard for now, because without it this PR fails for clusters with no etcd nodes and no cluster-wide offering. Once the same handling is in main I'll check it and remove the guard here, or rebase onto #14138 if it lands first.
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/utils/KubernetesClusterNodeCapacityReconciler.java:
##########
@@ -0,0 +1,68 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.kubernetes.cluster.utils;
+
+import java.io.File;
+
+import com.cloud.kubernetes.cluster.KubernetesCluster;
+import com.cloud.kubernetes.cluster.KubernetesServiceHelper.KubernetesClusterNodeType;
+import com.cloud.offering.ServiceOffering;
+import com.cloud.uservm.UserVm;
+
+/** Reconciles the guest and Kubernetes resource views after a live VM resize. */
+public interface KubernetesClusterNodeCapacityReconciler {
Review Comment:
Fair point. The interface is gone, it's one concrete class now, and the access and snapshot classes are top-level.
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/utils/KubernetesClusterNodeCapacityReconciler.java:
##########
@@ -0,0 +1,68 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.kubernetes.cluster.utils;
+
+import java.io.File;
+
+import com.cloud.kubernetes.cluster.KubernetesCluster;
+import com.cloud.kubernetes.cluster.KubernetesServiceHelper.KubernetesClusterNodeType;
+import com.cloud.offering.ServiceOffering;
+import com.cloud.uservm.UserVm;
+
+/** Reconciles the guest and Kubernetes resource views after a live VM resize. */
+public interface KubernetesClusterNodeCapacityReconciler {
+ NodeCapacitySnapshot captureBefore(KubernetesCluster cluster, UserVm vm, NodeAccess access);
+ boolean requiresKubeletRefresh(ServiceOffering oldOffering, ServiceOffering newOffering,
+ KubernetesClusterNodeType nodeType, KubernetesCluster.State originalState);
+ void cordonIfNeeded(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline);
+ void onlineGuestResources(NodeAccess access);
+ void verifyGuestResources(UserVm vm, ServiceOffering target, NodeCapacitySnapshot before, NodeAccess access, long deadline);
+ boolean isKubernetesResourcesCurrent(NodeCapacitySnapshot snapshot, ServiceOffering target);
+ void restartKubelet(UserVm vm, NodeAccess access, long deadline);
+ NodeCapacitySnapshot waitForKubernetesResources(KubernetesCluster cluster, UserVm vm, ServiceOffering target,
+ NodeCapacitySnapshot before, NodeAccess access, long deadline);
+ void restoreSchedulability(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline);
+
+ final class NodeAccess {
Review Comment:
Done for both classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "andrijapanicsb (via GitHub)" <gi...@apache.org>.
andrijapanicsb commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6469439294
@blueorangutan package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "ACSHomeBot (via GitHub) via commits" <co...@cloudstack.apache.org>.
ACSHomeBot commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6547752646
### Packaging results:
| Result | Artifact | Platform | Build Time |
|:--|:--|:--|--:|
| PASS | RPM | EL (EL8/9/10) | 29 min |
| PASS | DEB | Ubuntu, Debian | 21 min |
**Test packages are available at:**
- EL9/RPM: [https://packages.v2kmigrate.com/cloudstack/pr/14223/el9/](https://packages.v2kmigrate.com/cloudstack/pr/14223/el9/index.html)
- Debian/APT: [https://packages.v2kmigrate.com/cloudstack/pr/14223/debian/](https://packages.v2kmigrate.com/cloudstack/pr/14223/debian/index.html)
<!-- acshomebot-package-results:20261031T173610Z-0fbb49d0:1 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "andrijapanicsb (via GitHub) via commits" <co...@cloudstack.apache.org>.
andrijapanicsb commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6566984791
## Functional and regression testing on the final commit `a8a500f`
Lab: 2 management servers + 2 KVM hosts (AlmaLinux 9.1, libvirt 8.5.0 / QEMU 7.0.0), Kubernetes 1.33.1, CloudManaged cluster 1 control + 2 workers (3 after scale-out), isolated network. Packages built from `a8a500f`, identical management jar on both servers, new classes present. Hosts have 3 cores, so offerings went up to 3 vCPU and RAM moved in 256–512 MiB steps.
| Area | Tests | Result |
|---|---|---|
| Live increases: worker CPU-only, RAM-only, combined; control CPU-only (API) and RAM-only (UI); both node types in one call; cluster-wide offering; scale-out + offering change in one call | 13 | PASS |
| Jobs executed on either management server (node access and reconciliation from MGMT_B) | – | PASS |
| Same committed offering again, same-capacity offering change, stopped-cluster offering change | – | PASS: no node reads beyond one `get node`, no markers, no kubelet restart |
| Node preparation only after placement: control VM live-migrated for capacity, then node read, then the single preparation patch | – | PASS |
| Failure paths: control plane unreachable, node SSH unreachable, placement/capacity failure (no pending marker left), placement failure with earlier pending work (kept byte-identical), guest already above target, kubelet failure after resize (pending + CKS cordon kept, second worker untouched, retry finishes), scale timeout, operator cordon with failure, stale ownership marker, foreign marker, committed-offering retry with a mixed-offering node, VM stopped outside CKS, concurrent scale, parameter validation | 15 | PASS |
| Regression: scale-out, scale-in (`size` and `nodeids`), migrate after resize, stop/start, deleted offering in totals, plain (non-CKS) VM live/stopped scale, UI totals with mixed offerings | 7 | PASS |
| Autoscaling toggle | 1 | Pre-existing issue, same code on main → #14552 |
In every live resize: no VM reboot, no pod restart or UID change, DNS/HTTP service checks passed, the API server answered every readiness poll during control-node resizes, cluster metadata was committed only after all nodes succeeded.
Not tested: VPC tier / Netris and shared (direct-access) networks (node access changed in 306cc2e), multi-control HA, dedicated etcd, hypervisor rejection after dispatch (covered by unit tests), Kubernetes version upgrade after resizes.
Found during testing, pre-existing and independent of this PR:
- #14552: scale requests without an offering (cluster-autoscaler, `size`, enabling autoscaling) resize every node to an outdated cluster-wide offering
- #14553: a single CKS node can be scaled from the Instances view, leaving the cluster inconsistent
The Trillian smoke run on `a8a500f` is still to be done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-5775249023
## [Codecov](https://app.codecov.io/gh/apache/cloudstack/pull/14223?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 3.71%. Comparing base ([`602d9ec`](https://app.codecov.io/gh/apache/cloudstack/commit/602d9ec3e03e6350b55dd1349118ea2ccb766cca?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`6f6656f`](https://app.codecov.io/gh/apache/cloudstack/commit/6f6656f491717a9df02fee3e0f4951cb4821945c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
> :exclamation: There is a different number of reports uploaded between BASE (602d9ec) and HEAD (6f6656f). Click for more details.
>
> <details><summary>HEAD has 1 upload less than BASE</summary>
>
>| Flag | BASE (602d9ec) | HEAD (6f6656f) |
>|------|------|------|
>|unittests|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #14223 +/- ##
=============================================
- Coverage 19.91% 3.71% -16.21%
=============================================
Files 6373 487 -5886
Lines 577230 41992 -535238
Branches 70696 7942 -62754
=============================================
- Hits 114974 1558 -113416
+ Misses 449690 40208 -409482
+ Partials 12566 226 -12340
```
| [Flag](https://app.codecov.io/gh/apache/cloudstack/pull/14223/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [uitests](https://app.codecov.io/gh/apache/cloudstack/pull/14223/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `3.71% <ø> (ø)` | |
| [unittests](https://app.codecov.io/gh/apache/cloudstack/pull/14223/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/cloudstack/pull/14223?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (WIP)CKS: reconcile node capacity after live resize [cloudstack]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6300658003
## [](https://sonarcloud.io/dashboard?id=apache_cloudstack&pullRequest=14223) **Quality Gate failed**
Failed conditions
 [35.8% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_cloudstack&pullRequest=14223&metric=new_coverage&view=list) (required ≥ 40%)
<!-- sqra:qg_revision=ef4a5cf97fec7e3e902fa47d8a44408031a3d39f7e253abca74b80150002ebb1 -->
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_cloudstack&pullRequest=14223)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "sonarqubecloud[bot] (via GitHub) via commits" <co...@cloudstack.apache.org>.
sonarqubecloud[bot] commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6700565298
## [](https://sonarcloud.io/dashboard?id=apache_cloudstack&pullRequest=14223) **Quality Gate passed**
Issues
 [4 New issues](https://sonarcloud.io/project/issues?id=apache_cloudstack&pullRequest=14223&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_cloudstack&pullRequest=14223&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_cloudstack&pullRequest=14223&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [71.9% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_cloudstack&pullRequest=14223&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_cloudstack&pullRequest=14223&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_cloudstack&pullRequest=14223)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "ACSHomeBot (via GitHub) via commits" <co...@cloudstack.apache.org>.
ACSHomeBot commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6702544866
Accepted: `package kvm` for `c8e9058956aa`.
Planned: **RPM + DEB**, **KVM SystemVM**.
<!-- acshomebot-package-state:20261107T045940Z-ec53d659 -->
Status: **Queued** for **worker G**. Jobs ahead: **0**.
<!-- /acshomebot-package-state:20261107T045940Z-ec53d659 -->
<!-- acshomebot-package-status:20261107T045940Z-ec53d659 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (WIP)CKS: reconcile node capacity after live resize [cloudstack]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6293828685
## [](https://sonarcloud.io/dashboard?id=apache_cloudstack&pullRequest=14223) **Quality Gate failed**
Failed conditions
 [35.3% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_cloudstack&pullRequest=14223&metric=new_coverage&view=list) (required ≥ 40%)
<!-- sqra:qg_revision=42801e292da18711f56b32146363dee39e76203a630c4087a9c190a81bfca1ae -->
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_cloudstack&pullRequest=14223)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (WIP)CKS: reconcile node capacity after live resize [cloudstack]
Posted by "ACSHomeBot (via GitHub)" <gi...@apache.org>.
ACSHomeBot commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6295830072
Accepted: `package kvm G` for `2a404aa1ffcc`.
Planned: **RPM + DEB**, **KVM SystemVM**.
Status: **Queued** for **worker G**. Jobs ahead: **0**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "andrijapanicsb (via GitHub) via commits" <co...@cloudstack.apache.org>.
andrijapanicsb commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6567019687
@blueorangutan package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "sonarqubecloud[bot] (via GitHub) via commits" <co...@cloudstack.apache.org>.
sonarqubecloud[bot] commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6542636780
## [](https://sonarcloud.io/dashboard?id=apache_cloudstack&pullRequest=14223) **Quality Gate passed**
Issues
 [4 New issues](https://sonarcloud.io/project/issues?id=apache_cloudstack&pullRequest=14223&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_cloudstack&pullRequest=14223&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_cloudstack&pullRequest=14223&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [71.9% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_cloudstack&pullRequest=14223&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_cloudstack&pullRequest=14223&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_cloudstack&pullRequest=14223)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "ACSHomeBot (via GitHub)" <gi...@apache.org>.
ACSHomeBot commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6427490623
Accepted: `package kvm G` for `a4021f52ef3f`.
Planned: **RPM + DEB**, **KVM SystemVM**.
<!-- acshomebot-package-state:20261026T211454Z-04986cf3 -->
Status: **Queued** for **worker G**. Jobs ahead: **0**.
<!-- /acshomebot-package-state:20261026T211454Z-04986cf3 -->
<!-- acshomebot-package-status:20261026T211454Z-04986cf3 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "andrijapanicsb (via GitHub)" <gi...@apache.org>.
andrijapanicsb commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6427476988
@ACSHomeBot package G
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "andrijapanicsb (via GitHub)" <gi...@apache.org>.
andrijapanicsb commented on code in PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#discussion_r4394979028
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/utils/KubernetesClusterNodeCapacityReconcilerImpl.java:
##########
@@ -0,0 +1,232 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.kubernetes.cluster.utils;
+
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import com.cloud.kubernetes.cluster.KubernetesCluster;
+import com.cloud.kubernetes.cluster.KubernetesServiceHelper.KubernetesClusterNodeType;
+import com.cloud.offering.ServiceOffering;
+import com.cloud.uservm.UserVm;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.ssh.SshHelper;
+import com.google.gson.JsonArray;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParser;
+import org.apache.commons.lang3.StringUtils;
+
+/** SSH-backed implementation for a rolling CKS live-resize reconciliation. */
+public class KubernetesClusterNodeCapacityReconcilerImpl implements KubernetesClusterNodeCapacityReconciler {
+ // KVM QA observes 4,568,788 KiB in a 4,608 MiB guest after memory hotplug
+ // (about 146 MiB below the offering), so keep a small margin above that
+ // platform overhead while still rejecting materially undersized guests.
+ static final long MINIMUM_MEMORY_OVERHEAD_BYTES = 160L * 1024L * 1024L;
+ static final long KUBERNETES_MEMORY_REPORTING_TOLERANCE_BYTES = 16L * 1024L * 1024L;
+ private static final int COMMAND_TIMEOUT_MS = 30000;
+ private static final int POLL_INTERVAL_MS = 5000;
+ private static final Pattern NODE_NAME = Pattern.compile("[A-Za-z0-9][A-Za-z0-9.-]{0,252}");
+ private static final Pattern MEMORY_QUANTITY = Pattern.compile("([0-9]+)(Ki|Mi|Gi|Ti|K|M|G|T)?");
+ private static final String RESIZE_ANNOTATION = "cloudstack.apache.org/cks-live-resize";
+ static final String ONLINE_HOTPLUGGED_RESOURCES = "sudo sh -c 'for cpu in /sys/devices/system/cpu/cpu[0-9]*; do "
+ + "[ -e \"$cpu/online\" ] || continue; echo 1 > \"$cpu/online\"; done; "
+ + "for memory in /sys/devices/system/memory/memory[0-9]*; do "
+ + "[ -e \"$memory/online\" ] || continue; echo online > \"$memory/online\"; done'";
+
+ @Override
+ public boolean requiresKubeletRefresh(ServiceOffering oldOffering, ServiceOffering newOffering,
+ KubernetesClusterNodeType nodeType, KubernetesCluster.State originalState) {
+ return KubernetesCluster.State.Running == originalState
+ && (KubernetesClusterNodeType.WORKER == nodeType || KubernetesClusterNodeType.CONTROL == nodeType)
+ && capacityChanged(oldOffering, newOffering);
+ }
+
+ @Override
+ public NodeCapacitySnapshot captureBefore(KubernetesCluster cluster, UserVm vm, NodeAccess access) {
+ return nodeSnapshot(cluster, vm, access, guestResources(access));
+ }
+
+ @Override
+ public void cordonIfNeeded(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "=" + cluster.getUuid() + " --overwrite");
+ executeControl(access, "sudo /opt/bin/kubectl cordon " + node);
+ while (System.currentTimeMillis() < deadline) {
+ if (nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("CORDON", vm, "Kubernetes node did not become unschedulable");
+ }
+
+ @Override
+ public void onlineGuestResources(NodeAccess access) {
+ executeNode(access, ONLINE_HOTPLUGGED_RESOURCES);
+ }
+
+ @Override
+ public void verifyGuestResources(UserVm vm, ServiceOffering target, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ if (guestMatchesTarget(guestResources(access), target)) return;
+ sleep();
+ }
+ throw failure("GUEST_VERIFY", vm, "guest CPU or memory did not reach the target offering");
+ }
+
+ @Override
+ public boolean isKubernetesResourcesCurrent(NodeCapacitySnapshot snapshot, ServiceOffering target) {
+ GuestResources guest = new GuestResources(snapshot.getGuestOnlineCpuCount(), snapshot.getGuestMemoryKiB());
+ return guestMatchesTarget(guest, target) && kubernetesMatchesGuest(snapshot, snapshot);
+ }
+
+ @Override
+ public void restartKubelet(UserVm vm, NodeAccess access, long deadline) {
+ executeNode(access, "sudo systemctl restart kubelet");
+ while (System.currentTimeMillis() < deadline) {
+ if ("active".equals(executeNode(access, "sudo systemctl is-active kubelet").trim())) return;
+ sleep();
+ }
+ throw failure("KUBELET_RESTART", vm, "kubelet did not become active");
+ }
+
+ @Override
+ public NodeCapacitySnapshot waitForKubernetesResources(KubernetesCluster cluster, UserVm vm, ServiceOffering target,
+ NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ GuestResources guest = guestResources(access);
+ NodeCapacitySnapshot observed = nodeSnapshot(cluster, vm, access, guest);
+ if (guestMatchesTarget(guest, target) && kubernetesMatchesGuest(observed, before)) return observed;
+ sleep();
+ }
+ throw failure("CAPACITY_VERIFY", vm, "Kubernetes capacity did not match the resized guest");
+ }
+
+ @Override
+ public void restoreSchedulability(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl uncordon " + node);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "-");
+ while (System.currentTimeMillis() < deadline) {
+ if (!nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("RESTORE_SCHEDULABILITY", vm, "Kubernetes node remained unschedulable");
+ }
+
+ public static boolean capacityChanged(ServiceOffering oldOffering, ServiceOffering targetOffering) {
+ return oldOffering != null && targetOffering != null
+ && (!Objects.equals(oldOffering.getCpu(), targetOffering.getCpu())
+ || !Objects.equals(oldOffering.getRamSize(), targetOffering.getRamSize()));
+ }
+
+ static long parseCpuMillis(String value) { return value.endsWith("m") ? Long.parseLong(value.substring(0, value.length() - 1)) : Long.parseLong(value) * 1000L; }
+ static long parseMemoryBytes(String value) {
+ Matcher matcher = MEMORY_QUANTITY.matcher(value);
+ if (!matcher.matches()) throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ long number = Long.parseLong(matcher.group(1)); String unit = matcher.group(2);
+ if (unit == null) return number;
+ switch (unit) {
+ case "Ki": return number * 1024L; case "Mi": return number * 1024L * 1024L; case "Gi": return number * 1024L * 1024L * 1024L;
+ case "Ti": return number * 1024L * 1024L * 1024L * 1024L; case "K": return number * 1000L; case "M": return number * 1000L * 1000L;
+ case "G": return number * 1000L * 1000L * 1000L; case "T": return number * 1000L * 1000L * 1000L * 1000L;
+ default: throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ }
+ }
+
+ private NodeCapacitySnapshot nodeSnapshot(KubernetesCluster cluster, UserVm vm, NodeAccess access, GuestResources guest) {
+ JsonObject node = new JsonParser().parse(executeControl(access, "sudo /opt/bin/kubectl get node " + nodeName(vm) + " -o json")).getAsJsonObject();
+ JsonObject annotations = node.getAsJsonObject("metadata").has("annotations") ? node.getAsJsonObject("metadata").getAsJsonObject("annotations") : null;
+ boolean cksCordon = annotations != null && annotations.has(RESIZE_ANNOTATION) && cluster.getUuid().equals(annotations.get(RESIZE_ANNOTATION).getAsString());
+ JsonObject spec = node.has("spec") ? node.getAsJsonObject("spec") : new JsonObject(); JsonObject status = node.getAsJsonObject("status");
+ JsonObject capacity = status.getAsJsonObject("capacity"); JsonObject allocatable = status.getAsJsonObject("allocatable");
+ return new NodeCapacitySnapshot(spec.has("unschedulable") && spec.get("unschedulable").getAsBoolean(), cksCordon,
+ guest == null ? 0 : guest.cpu, guest == null ? 0 : guest.memoryKiB, parseCpuMillis(capacity.get("cpu").getAsString()),
+ parseMemoryBytes(capacity.get("memory").getAsString()), parseCpuMillis(allocatable.get("cpu").getAsString()),
+ parseMemoryBytes(allocatable.get("memory").getAsString()), isReady(status));
+ }
+
+ private boolean isReady(JsonObject status) {
+ JsonArray conditions = status.getAsJsonArray("conditions");
+ for (JsonElement condition : conditions) { JsonObject item = condition.getAsJsonObject(); if ("Ready".equals(item.get("type").getAsString()) && "True".equals(item.get("status").getAsString())) return true; }
+ return false;
+ }
+
+ private GuestResources guestResources(NodeAccess access) {
+ String[] values = executeNode(access, "getconf _NPROCESSORS_ONLN; awk '/^MemTotal:/ {print $2}' /proc/meminfo").trim().split("\\s+");
+ if (values.length != 2) throw new CloudRuntimeException("Unable to read guest CPU and memory");
+ return new GuestResources(Long.parseLong(values[0]), Long.parseLong(values[1]));
+ }
+
+ private boolean guestMatchesTarget(GuestResources observed, ServiceOffering target) {
+ long targetMemory = target.getRamSize() * 1024L * 1024L; long observedMemory = observed.memoryKiB * 1024L;
+ long overhead = Math.max(targetMemory / 50L, MINIMUM_MEMORY_OVERHEAD_BYTES);
+ return observed.cpu == target.getCpu() && observedMemory <= targetMemory && targetMemory - observedMemory <= overhead;
+ }
+
+ private boolean kubernetesMatchesGuest(NodeCapacitySnapshot observed, NodeCapacitySnapshot before) {
+ return observed.isReady() && observed.getCapacityCpuMillis() == observed.getGuestOnlineCpuCount() * 1000L
+ && Math.abs(observed.getCapacityMemoryBytes() - observed.getGuestMemoryKiB() * 1024L) <= KUBERNETES_MEMORY_REPORTING_TOLERANCE_BYTES
+ && (observed.getGuestOnlineCpuCount() <= before.getGuestOnlineCpuCount() || observed.getAllocatableCpuMillis() > before.getAllocatableCpuMillis())
+ && (observed.getGuestMemoryKiB() <= before.getGuestMemoryKiB() || observed.getAllocatableMemoryBytes() > before.getAllocatableMemoryBytes());
+ }
+
+ protected String executeControl(NodeAccess access, String command) {
+ return execute(access.getControlAddress(), access.getControlPort(), access, command);
+ }
+
+ protected String executeNode(NodeAccess access, String command) {
+ return execute(access.getNodeAddress(), access.getNodePort(), access, command);
+ }
+
+ private String execute(String address, int port, NodeAccess access, String command) {
+ Pair<Boolean, String> result;
+ try {
+ result = SshHelper.sshExecute(address, port, access.getUser(), access.getSshKeyFile(), null, command, 10000, 10000, COMMAND_TIMEOUT_MS);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new NodeCapacityReconciliationException("CKS live-resize command interrupted", e);
+ } catch (RuntimeException e) {
+ throw e;
+ } catch (Exception e) {
+ // SshHelper declares Exception, including authentication failures.
+ // Translate only at this legacy boundary, not around the resize workflow.
+ throw new NodeCapacityReconciliationException("CKS live-resize SSH command failed", e);
+ }
+ if (!Boolean.TRUE.equals(result.first())) {
+ throw new NodeCapacityReconciliationException("CKS live-resize command failed");
+ }
+ return StringUtils.defaultString(result.second());
+ }
+ private String nodeName(UserVm vm) { String name = StringUtils.lowerCase(vm.getHostName()); if (!NODE_NAME.matcher(name).matches()) throw new CloudRuntimeException("Invalid Kubernetes node name for live resize"); return name; }
Review Comment:
Done, a blank host name now fails with a clear message before any command runs.
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/utils/KubernetesClusterNodeCapacityReconcilerImpl.java:
##########
@@ -0,0 +1,232 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.kubernetes.cluster.utils;
+
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import com.cloud.kubernetes.cluster.KubernetesCluster;
+import com.cloud.kubernetes.cluster.KubernetesServiceHelper.KubernetesClusterNodeType;
+import com.cloud.offering.ServiceOffering;
+import com.cloud.uservm.UserVm;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.ssh.SshHelper;
+import com.google.gson.JsonArray;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParser;
+import org.apache.commons.lang3.StringUtils;
+
+/** SSH-backed implementation for a rolling CKS live-resize reconciliation. */
+public class KubernetesClusterNodeCapacityReconcilerImpl implements KubernetesClusterNodeCapacityReconciler {
+ // KVM QA observes 4,568,788 KiB in a 4,608 MiB guest after memory hotplug
+ // (about 146 MiB below the offering), so keep a small margin above that
+ // platform overhead while still rejecting materially undersized guests.
+ static final long MINIMUM_MEMORY_OVERHEAD_BYTES = 160L * 1024L * 1024L;
+ static final long KUBERNETES_MEMORY_REPORTING_TOLERANCE_BYTES = 16L * 1024L * 1024L;
+ private static final int COMMAND_TIMEOUT_MS = 30000;
+ private static final int POLL_INTERVAL_MS = 5000;
+ private static final Pattern NODE_NAME = Pattern.compile("[A-Za-z0-9][A-Za-z0-9.-]{0,252}");
+ private static final Pattern MEMORY_QUANTITY = Pattern.compile("([0-9]+)(Ki|Mi|Gi|Ti|K|M|G|T)?");
+ private static final String RESIZE_ANNOTATION = "cloudstack.apache.org/cks-live-resize";
+ static final String ONLINE_HOTPLUGGED_RESOURCES = "sudo sh -c 'for cpu in /sys/devices/system/cpu/cpu[0-9]*; do "
+ + "[ -e \"$cpu/online\" ] || continue; echo 1 > \"$cpu/online\"; done; "
+ + "for memory in /sys/devices/system/memory/memory[0-9]*; do "
+ + "[ -e \"$memory/online\" ] || continue; echo online > \"$memory/online\"; done'";
+
+ @Override
+ public boolean requiresKubeletRefresh(ServiceOffering oldOffering, ServiceOffering newOffering,
+ KubernetesClusterNodeType nodeType, KubernetesCluster.State originalState) {
+ return KubernetesCluster.State.Running == originalState
+ && (KubernetesClusterNodeType.WORKER == nodeType || KubernetesClusterNodeType.CONTROL == nodeType)
+ && capacityChanged(oldOffering, newOffering);
+ }
+
+ @Override
+ public NodeCapacitySnapshot captureBefore(KubernetesCluster cluster, UserVm vm, NodeAccess access) {
+ return nodeSnapshot(cluster, vm, access, guestResources(access));
+ }
+
+ @Override
+ public void cordonIfNeeded(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "=" + cluster.getUuid() + " --overwrite");
+ executeControl(access, "sudo /opt/bin/kubectl cordon " + node);
+ while (System.currentTimeMillis() < deadline) {
+ if (nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("CORDON", vm, "Kubernetes node did not become unschedulable");
+ }
+
+ @Override
+ public void onlineGuestResources(NodeAccess access) {
+ executeNode(access, ONLINE_HOTPLUGGED_RESOURCES);
+ }
+
+ @Override
+ public void verifyGuestResources(UserVm vm, ServiceOffering target, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ if (guestMatchesTarget(guestResources(access), target)) return;
+ sleep();
+ }
+ throw failure("GUEST_VERIFY", vm, "guest CPU or memory did not reach the target offering");
+ }
+
+ @Override
+ public boolean isKubernetesResourcesCurrent(NodeCapacitySnapshot snapshot, ServiceOffering target) {
+ GuestResources guest = new GuestResources(snapshot.getGuestOnlineCpuCount(), snapshot.getGuestMemoryKiB());
+ return guestMatchesTarget(guest, target) && kubernetesMatchesGuest(snapshot, snapshot);
+ }
+
+ @Override
+ public void restartKubelet(UserVm vm, NodeAccess access, long deadline) {
+ executeNode(access, "sudo systemctl restart kubelet");
+ while (System.currentTimeMillis() < deadline) {
+ if ("active".equals(executeNode(access, "sudo systemctl is-active kubelet").trim())) return;
+ sleep();
+ }
+ throw failure("KUBELET_RESTART", vm, "kubelet did not become active");
+ }
+
+ @Override
+ public NodeCapacitySnapshot waitForKubernetesResources(KubernetesCluster cluster, UserVm vm, ServiceOffering target,
+ NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ GuestResources guest = guestResources(access);
+ NodeCapacitySnapshot observed = nodeSnapshot(cluster, vm, access, guest);
+ if (guestMatchesTarget(guest, target) && kubernetesMatchesGuest(observed, before)) return observed;
+ sleep();
+ }
+ throw failure("CAPACITY_VERIFY", vm, "Kubernetes capacity did not match the resized guest");
+ }
+
+ @Override
+ public void restoreSchedulability(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl uncordon " + node);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "-");
+ while (System.currentTimeMillis() < deadline) {
+ if (!nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("RESTORE_SCHEDULABILITY", vm, "Kubernetes node remained unschedulable");
+ }
+
+ public static boolean capacityChanged(ServiceOffering oldOffering, ServiceOffering targetOffering) {
+ return oldOffering != null && targetOffering != null
+ && (!Objects.equals(oldOffering.getCpu(), targetOffering.getCpu())
+ || !Objects.equals(oldOffering.getRamSize(), targetOffering.getRamSize()));
+ }
+
+ static long parseCpuMillis(String value) { return value.endsWith("m") ? Long.parseLong(value.substring(0, value.length() - 1)) : Long.parseLong(value) * 1000L; }
+ static long parseMemoryBytes(String value) {
+ Matcher matcher = MEMORY_QUANTITY.matcher(value);
+ if (!matcher.matches()) throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ long number = Long.parseLong(matcher.group(1)); String unit = matcher.group(2);
+ if (unit == null) return number;
+ switch (unit) {
+ case "Ki": return number * 1024L; case "Mi": return number * 1024L * 1024L; case "Gi": return number * 1024L * 1024L * 1024L;
+ case "Ti": return number * 1024L * 1024L * 1024L * 1024L; case "K": return number * 1000L; case "M": return number * 1000L * 1000L;
+ case "G": return number * 1000L * 1000L * 1000L; case "T": return number * 1000L * 1000L * 1000L * 1000L;
+ default: throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ }
+ }
+
+ private NodeCapacitySnapshot nodeSnapshot(KubernetesCluster cluster, UserVm vm, NodeAccess access, GuestResources guest) {
+ JsonObject node = new JsonParser().parse(executeControl(access, "sudo /opt/bin/kubectl get node " + nodeName(vm) + " -o json")).getAsJsonObject();
+ JsonObject annotations = node.getAsJsonObject("metadata").has("annotations") ? node.getAsJsonObject("metadata").getAsJsonObject("annotations") : null;
+ boolean cksCordon = annotations != null && annotations.has(RESIZE_ANNOTATION) && cluster.getUuid().equals(annotations.get(RESIZE_ANNOTATION).getAsString());
+ JsonObject spec = node.has("spec") ? node.getAsJsonObject("spec") : new JsonObject(); JsonObject status = node.getAsJsonObject("status");
+ JsonObject capacity = status.getAsJsonObject("capacity"); JsonObject allocatable = status.getAsJsonObject("allocatable");
+ return new NodeCapacitySnapshot(spec.has("unschedulable") && spec.get("unschedulable").getAsBoolean(), cksCordon,
+ guest == null ? 0 : guest.cpu, guest == null ? 0 : guest.memoryKiB, parseCpuMillis(capacity.get("cpu").getAsString()),
+ parseMemoryBytes(capacity.get("memory").getAsString()), parseCpuMillis(allocatable.get("cpu").getAsString()),
+ parseMemoryBytes(allocatable.get("memory").getAsString()), isReady(status));
+ }
+
+ private boolean isReady(JsonObject status) {
+ JsonArray conditions = status.getAsJsonArray("conditions");
+ for (JsonElement condition : conditions) { JsonObject item = condition.getAsJsonObject(); if ("Ready".equals(item.get("type").getAsString()) && "True".equals(item.get("status").getAsString())) return true; }
+ return false;
+ }
+
+ private GuestResources guestResources(NodeAccess access) {
+ String[] values = executeNode(access, "getconf _NPROCESSORS_ONLN; awk '/^MemTotal:/ {print $2}' /proc/meminfo").trim().split("\\s+");
+ if (values.length != 2) throw new CloudRuntimeException("Unable to read guest CPU and memory");
+ return new GuestResources(Long.parseLong(values[0]), Long.parseLong(values[1]));
+ }
+
+ private boolean guestMatchesTarget(GuestResources observed, ServiceOffering target) {
+ long targetMemory = target.getRamSize() * 1024L * 1024L; long observedMemory = observed.memoryKiB * 1024L;
+ long overhead = Math.max(targetMemory / 50L, MINIMUM_MEMORY_OVERHEAD_BYTES);
+ return observed.cpu == target.getCpu() && observedMemory <= targetMemory && targetMemory - observedMemory <= overhead;
+ }
+
+ private boolean kubernetesMatchesGuest(NodeCapacitySnapshot observed, NodeCapacitySnapshot before) {
+ return observed.isReady() && observed.getCapacityCpuMillis() == observed.getGuestOnlineCpuCount() * 1000L
+ && Math.abs(observed.getCapacityMemoryBytes() - observed.getGuestMemoryKiB() * 1024L) <= KUBERNETES_MEMORY_REPORTING_TOLERANCE_BYTES
+ && (observed.getGuestOnlineCpuCount() <= before.getGuestOnlineCpuCount() || observed.getAllocatableCpuMillis() > before.getAllocatableCpuMillis())
+ && (observed.getGuestMemoryKiB() <= before.getGuestMemoryKiB() || observed.getAllocatableMemoryBytes() > before.getAllocatableMemoryBytes());
+ }
+
+ protected String executeControl(NodeAccess access, String command) {
+ return execute(access.getControlAddress(), access.getControlPort(), access, command);
+ }
+
+ protected String executeNode(NodeAccess access, String command) {
+ return execute(access.getNodeAddress(), access.getNodePort(), access, command);
+ }
+
+ private String execute(String address, int port, NodeAccess access, String command) {
+ Pair<Boolean, String> result;
+ try {
+ result = SshHelper.sshExecute(address, port, access.getUser(), access.getSshKeyFile(), null, command, 10000, 10000, COMMAND_TIMEOUT_MS);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new NodeCapacityReconciliationException("CKS live-resize command interrupted", e);
+ } catch (RuntimeException e) {
+ throw e;
+ } catch (Exception e) {
+ // SshHelper declares Exception, including authentication failures.
+ // Translate only at this legacy boundary, not around the resize workflow.
+ throw new NodeCapacityReconciliationException("CKS live-resize SSH command failed", e);
+ }
+ if (!Boolean.TRUE.equals(result.first())) {
+ throw new NodeCapacityReconciliationException("CKS live-resize command failed");
+ }
+ return StringUtils.defaultString(result.second());
+ }
+ private String nodeName(UserVm vm) { String name = StringUtils.lowerCase(vm.getHostName()); if (!NODE_NAME.matcher(name).matches()) throw new CloudRuntimeException("Invalid Kubernetes node name for live resize"); return name; }
+ private NodeCapacityReconciliationException failure(String phase, UserVm vm, String message) {
+ return new NodeCapacityReconciliationException("CKS live resize " + phase + " failed for VM " + vm.getUuid() + ": " + message);
+ }
+
+ private void sleep() {
+ try {
+ Thread.sleep(POLL_INTERVAL_MS);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new NodeCapacityReconciliationException("CKS live-resize resource polling interrupted", e);
+ }
+ }
+ private static class GuestResources { private final long cpu; private final long memoryKiB; GuestResources(long cpu, long memoryKiB) { this.cpu = cpu; this.memoryKiB = memoryKiB; } }
Review Comment:
Done here as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6427271112
## [](https://sonarcloud.io/dashboard?id=apache_cloudstack&pullRequest=14223) **Quality Gate passed**
Issues
 [30 New issues](https://sonarcloud.io/project/issues?id=apache_cloudstack&pullRequest=14223&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_cloudstack&pullRequest=14223&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_cloudstack&pullRequest=14223&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [67.4% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_cloudstack&pullRequest=14223&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_cloudstack&pullRequest=14223&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_cloudstack&pullRequest=14223)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "ACSHomeBot (via GitHub)" <gi...@apache.org>.
ACSHomeBot commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6473138495
### Packaging results:
| Result | Artifact | Platform | Build Time |
|:--|:--|:--|--:|
| PASS | RPM | EL (EL8/9/10) | 34 min |
| PASS | DEB | Ubuntu, Debian | 23 min |
**Test packages are available at:**
- EL9/RPM: [https://packages.v2kmigrate.com/cloudstack/pr/14223/el9/](https://packages.v2kmigrate.com/cloudstack/pr/14223/el9/index.html)
- Debian/APT: [https://packages.v2kmigrate.com/cloudstack/pr/14223/debian/](https://packages.v2kmigrate.com/cloudstack/pr/14223/debian/index.html)
<!-- acshomebot-package-results:20261028T135941Z-787a23b0:2 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (WIP)CKS: reconcile node capacity after live resize [cloudstack]
Posted by "ACSHomeBot (via GitHub)" <gi...@apache.org>.
ACSHomeBot commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-5775851934
Accepted: `package kvm G` for `d512edb7e008`.
Planned: **RPM + DEB**, **KVM SystemVM**.
Status: **Queued** for **worker G**. Jobs ahead: **0**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6504435912
@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "andrijapanicsb (via GitHub)" <gi...@apache.org>.
andrijapanicsb commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6455363859
Hi @Pearl1594, thanks a lot for the detailed review, I really appreciate the time you put into it. I've pushed the changes in 306cc2e02f and replied to each of your comments above.
While working through them I also found two problems I had missed: a node cordoned by an operator would never have been retried after a failure, and the node SSH address was wrong on Netris VPC tiers. Both are fixed now. In two places I went a slightly different way from what you suggested (the etcd check and the skip on SSH failure), and I explained why in those threads.
Let me know if anything still looks off.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "ACSHomeBot (via GitHub)" <gi...@apache.org>.
ACSHomeBot commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6470024570
### Packaging results:
| Result | Artifact | Platform | Build Time |
|:--|:--|:--|--:|
| PASS | RPM | EL (EL8/9/10) | 35 min |
| PASS | DEB | Ubuntu, Debian | 22 min |
**Test packages are available at:**
- EL9/RPM: [https://packages.v2kmigrate.com/cloudstack/pr/14223/el9/](https://packages.v2kmigrate.com/cloudstack/pr/14223/el9/index.html)
- Debian/APT: [https://packages.v2kmigrate.com/cloudstack/pr/14223/debian/](https://packages.v2kmigrate.com/cloudstack/pr/14223/debian/index.html)
<!-- acshomebot-package-results:20261028T121530Z-028d0fb0:1 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "ACSHomeBot (via GitHub)" <gi...@apache.org>.
ACSHomeBot commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6468718361
Accepted: `package kvm G` for `306cc2e02f7b`.
Planned: **RPM + DEB**, **KVM SystemVM**.
<!-- acshomebot-package-state:20261028T121530Z-028d0fb0 -->
Status: **Queued** for **worker G**. Jobs ahead: **0**.
<!-- /acshomebot-package-state:20261028T121530Z-028d0fb0 -->
<!-- acshomebot-package-status:20261028T121530Z-028d0fb0 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (WIP)CKS: reconcile node capacity after live resize [cloudstack]
Posted by "ACSHomeBot (via GitHub)" <gi...@apache.org>.
ACSHomeBot commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6322893981
Accepted: `package kvm G` for `f92e52d736b9`.
Planned: **RPM + DEB**, **KVM SystemVM**.
Status: **Queued** for **worker G**. Jobs ahead: **0**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "ACSHomeBot (via GitHub)" <gi...@apache.org>.
ACSHomeBot commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6428679081
### Packaging results:
| Result | Artifact | Platform | Build Time |
|:--|:--|:--|--:|
| PASS | RPM | EL (EL8/9/10) | 36 min |
| PASS | DEB | Ubuntu, Debian | 23 min |
**Test packages are available at:**
- EL9/RPM: [https://packages.v2kmigrate.com/cloudstack/pr/14223/el9/](https://packages.v2kmigrate.com/cloudstack/pr/14223/el9/index.html)
- Debian/APT: [https://packages.v2kmigrate.com/cloudstack/pr/14223/debian/](https://packages.v2kmigrate.com/cloudstack/pr/14223/debian/index.html)
<!-- acshomebot-package-results:20261026T211454Z-04986cf3:1 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6411354872
@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6326623451
@andrijapanicsb a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "blueorangutan (via GitHub) via commits" <co...@cloudstack.apache.org>.
blueorangutan commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6706149682
@andrijapanicsb a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (WIP)CKS: reconcile node capacity after live resize [cloudstack]
Posted by "ACSHomeBot (via GitHub)" <gi...@apache.org>.
ACSHomeBot commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6296978428
### Packaging results:
| Result | Artifact | Platform | Build Time |
|:--|:--|:--|--:|
| PASS | RPM | EL (EL8/9/10) | 31 min |
| PASS | DEB | Ubuntu, Debian | 22 min |
**Test packages are available at:**
- EL9/RPM: [https://packages.v2kmigrate.com/cloudstack/pr/14223/el9/](https://packages.v2kmigrate.com/cloudstack/pr/14223/el9/index.html)
- Debian/APT: [https://packages.v2kmigrate.com/cloudstack/pr/14223/debian/](https://packages.v2kmigrate.com/cloudstack/pr/14223/debian/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (WIP)CKS: reconcile node capacity after live resize [cloudstack]
Posted by "andrijapanicsb (via GitHub)" <gi...@apache.org>.
andrijapanicsb commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6295811091
@ACSHomeBot package G
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (WIP)CKS: reconcile node capacity after live resize [cloudstack]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6321316545
## [](https://sonarcloud.io/dashboard?id=apache_cloudstack&pullRequest=14223) **Quality Gate passed**
Issues
 [27 New issues](https://sonarcloud.io/project/issues?id=apache_cloudstack&pullRequest=14223&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_cloudstack&pullRequest=14223&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_cloudstack&pullRequest=14223&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [48.5% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_cloudstack&pullRequest=14223&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_cloudstack&pullRequest=14223&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_cloudstack&pullRequest=14223)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "andrijapanicsb (via GitHub)" <gi...@apache.org>.
andrijapanicsb commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6425744358
The technical findings are fair, and I’ve addressed them. The remark about instructing “my AI”, however, is unnecessary. Please keep the review focused on concrete issues in the code.
I’m contributing here as an operator because the existing live-resize path could successfully resize a VM while Kubernetes continued to advertise the old resources. Whatever the original implementation’s scope, operators still had to complete the Kubernetes-side work manually. A successful VM resize is not, by itself, a successful Kubernetes resize.
There is some irony in focusing on a non-developer’s use of AI when that contribution is closing this end-to-end gap. Clean coding principles are a fair requirement; complete behaviour and end-to-end testing deserve the same scrutiny. Please judge the implementation and its verification—not the contributor’s background or authoring tools.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (WIP)CKS: reconcile node capacity after live resize [cloudstack]
Posted by "ACSHomeBot (via GitHub)" <gi...@apache.org>.
ACSHomeBot commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6292097005
Accepted: `package kvm G` for `548011910846`.
Planned: **RPM + DEB**, **KVM SystemVM**.
Status: **Queued** for **worker G**. Jobs ahead: **0**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "blueorangutan (via GitHub) via commits" <co...@cloudstack.apache.org>.
blueorangutan commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6567041532
@andrijapanicsb a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on code in PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#discussion_r4398519493
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterScaleWorker.java:
##########
@@ -361,7 +361,7 @@ private void validateKubernetesClusterScaleSizeParameters(KubernetesClusterNodeT
}
}
- private void scaleKubernetesClusterOffering(KubernetesClusterNodeType nodeType, ServiceOffering serviceOffering,
+ protected void scaleKubernetesClusterOffering(KubernetesClusterNodeType nodeType, ServiceOffering serviceOffering,
Review Comment:
@andrija, looks like this one has gone with your changes on Pearl's remarks. But have a look at these: https://github.com/apache/cloudstack/pull/14223/changes?panel=alerts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "andrijapanicsb (via GitHub)" <gi...@apache.org>.
andrijapanicsb commented on code in PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#discussion_r4394969622
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterScaleWorker.java:
##########
@@ -229,32 +236,36 @@ private KubernetesClusterVO updateKubernetesClusterEntryForNodeType(final Long n
}
protected Pair<Long, Long> calculateNewClusterCountAndCapacity(Long newWorkerSize, KubernetesClusterNodeType nodeType, ServiceOffering serviceOffering) {
- long cores;
- long memory;
- long totalClusterSize = newWorkerSize == null ? kubernetesCluster.getTotalNodeCount() : (newWorkerSize + kubernetesCluster.getControlNodeCount() + kubernetesCluster.getEtcdNodeCount());
-
- if (nodeType == DEFAULT) {
- cores = serviceOffering.getCpu() * totalClusterSize;
- memory = serviceOffering.getRamSize() * totalClusterSize;
- } else {
- long nodeCount = getNodeCountForType(nodeType, kubernetesCluster);
- Long existingOfferingId = getExistingOfferingIdForNodeType(nodeType, kubernetesCluster);
- if (existingOfferingId == null) {
- existingOfferingId = serviceOffering.getId();
- }
- ServiceOfferingVO previousOffering = serviceOfferingDao.findById(existingOfferingId);
- Pair<Long, Long> previousNodesCapacity = calculateNodesCapacity(previousOffering, nodeCount);
- if (WORKER == nodeType) {
- nodeCount = newWorkerSize == null ? kubernetesCluster.getNodeCount() : newWorkerSize;
+ long cores = 0;
+ long memory = 0;
+ // VM offerings have already changed here. Recompute the total from the
+ // configured node types instead of subtracting a supposedly old VM offering.
+ for (KubernetesClusterNodeType type : List.of(CONTROL, WORKER, ETCD)) {
Review Comment:
Done, the capacity sum now uses `CLUSTER_NODES_TYPES_LIST`. I kept the CONTROL, ETCD, WORKER order in the scale loop itself, since the order matters there.
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterScaleWorker.java:
##########
@@ -229,32 +236,36 @@ private KubernetesClusterVO updateKubernetesClusterEntryForNodeType(final Long n
}
protected Pair<Long, Long> calculateNewClusterCountAndCapacity(Long newWorkerSize, KubernetesClusterNodeType nodeType, ServiceOffering serviceOffering) {
- long cores;
- long memory;
- long totalClusterSize = newWorkerSize == null ? kubernetesCluster.getTotalNodeCount() : (newWorkerSize + kubernetesCluster.getControlNodeCount() + kubernetesCluster.getEtcdNodeCount());
-
- if (nodeType == DEFAULT) {
- cores = serviceOffering.getCpu() * totalClusterSize;
- memory = serviceOffering.getRamSize() * totalClusterSize;
- } else {
- long nodeCount = getNodeCountForType(nodeType, kubernetesCluster);
- Long existingOfferingId = getExistingOfferingIdForNodeType(nodeType, kubernetesCluster);
- if (existingOfferingId == null) {
- existingOfferingId = serviceOffering.getId();
- }
- ServiceOfferingVO previousOffering = serviceOfferingDao.findById(existingOfferingId);
- Pair<Long, Long> previousNodesCapacity = calculateNodesCapacity(previousOffering, nodeCount);
- if (WORKER == nodeType) {
- nodeCount = newWorkerSize == null ? kubernetesCluster.getNodeCount() : newWorkerSize;
+ long cores = 0;
+ long memory = 0;
+ // VM offerings have already changed here. Recompute the total from the
+ // configured node types instead of subtracting a supposedly old VM offering.
+ for (KubernetesClusterNodeType type : List.of(CONTROL, WORKER, ETCD)) {
+ long nodeCount = WORKER == type && newWorkerSize != null ? newWorkerSize : getNodeCountForType(type, kubernetesCluster);
+ if (nodeCount == 0) {
+ continue;
}
- Pair<Long, Long> newNodesCapacity = calculateNodesCapacity(serviceOffering, nodeCount);
- Pair<Long, Long> newClusterCapacity = calculateClusterNewCapacity(kubernetesCluster, previousNodesCapacity, newNodesCapacity);
- cores = newClusterCapacity.first();
- memory = newClusterCapacity.second();
+ ServiceOffering offering = nodeType == DEFAULT || nodeType == type ? serviceOffering :
+ serviceOfferingDao.findByIdIncludingRemoved(getConfiguredOfferingIdForNodeType(type));
Review Comment:
Done. The per-type to cluster fallback is in one place now (`getServiceOfferingIdForNodeTypeOnCluster`), and `getServiceOfferingForNodeTypeOnCluster` uses it.
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterScaleWorker.java:
##########
@@ -229,32 +236,36 @@ private KubernetesClusterVO updateKubernetesClusterEntryForNodeType(final Long n
}
protected Pair<Long, Long> calculateNewClusterCountAndCapacity(Long newWorkerSize, KubernetesClusterNodeType nodeType, ServiceOffering serviceOffering) {
- long cores;
- long memory;
- long totalClusterSize = newWorkerSize == null ? kubernetesCluster.getTotalNodeCount() : (newWorkerSize + kubernetesCluster.getControlNodeCount() + kubernetesCluster.getEtcdNodeCount());
-
- if (nodeType == DEFAULT) {
- cores = serviceOffering.getCpu() * totalClusterSize;
- memory = serviceOffering.getRamSize() * totalClusterSize;
- } else {
- long nodeCount = getNodeCountForType(nodeType, kubernetesCluster);
- Long existingOfferingId = getExistingOfferingIdForNodeType(nodeType, kubernetesCluster);
- if (existingOfferingId == null) {
- existingOfferingId = serviceOffering.getId();
- }
- ServiceOfferingVO previousOffering = serviceOfferingDao.findById(existingOfferingId);
- Pair<Long, Long> previousNodesCapacity = calculateNodesCapacity(previousOffering, nodeCount);
- if (WORKER == nodeType) {
- nodeCount = newWorkerSize == null ? kubernetesCluster.getNodeCount() : newWorkerSize;
+ long cores = 0;
+ long memory = 0;
+ // VM offerings have already changed here. Recompute the total from the
+ // configured node types instead of subtracting a supposedly old VM offering.
+ for (KubernetesClusterNodeType type : List.of(CONTROL, WORKER, ETCD)) {
+ long nodeCount = WORKER == type && newWorkerSize != null ? newWorkerSize : getNodeCountForType(type, kubernetesCluster);
+ if (nodeCount == 0) {
+ continue;
}
- Pair<Long, Long> newNodesCapacity = calculateNodesCapacity(serviceOffering, nodeCount);
- Pair<Long, Long> newClusterCapacity = calculateClusterNewCapacity(kubernetesCluster, previousNodesCapacity, newNodesCapacity);
- cores = newClusterCapacity.first();
- memory = newClusterCapacity.second();
+ ServiceOffering offering = nodeType == DEFAULT || nodeType == type ? serviceOffering :
+ serviceOfferingDao.findByIdIncludingRemoved(getConfiguredOfferingIdForNodeType(type));
+ Pair<Long, Long> capacity = calculateNodesCapacity(offering, nodeCount);
+ cores += capacity.first();
+ memory += capacity.second();
}
return new Pair<>(cores, memory);
}
+ private Long getConfiguredOfferingIdForNodeType(KubernetesClusterNodeType nodeType) {
+ Long offeringId = null;
+ if (WORKER == nodeType) {
+ offeringId = kubernetesCluster.getWorkerNodeServiceOfferingId();
+ } else if (CONTROL == nodeType) {
+ offeringId = kubernetesCluster.getControlNodeServiceOfferingId();
+ } else if (ETCD == nodeType) {
+ offeringId = kubernetesCluster.getEtcdNodeServiceOfferingId();
+ }
+ return offeringId == null ? kubernetesCluster.getServiceOfferingId() : offeringId;
+ }
Review Comment:
Agreed, the copy is gone. One difference: for the VM's current offering and for the cluster totals I now look the offering up including removed ones, so the totals no longer break when an offering has been deleted. Other callers keep their current lookup. A missing offering ID now fails with a clear message, and the offerings of all node types in the request are checked before any VM is touched.
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterScaleWorker.java:
##########
@@ -376,24 +377,171 @@ private void scaleKubernetesClusterOffering(KubernetesClusterNodeType nodeType,
List<KubernetesClusterVmMapVO> vmList = kubernetesClusterVmMapDao.listByClusterIdAndVmType(kubernetesCluster.getId(), nodeType);
final long tobeScaledVMCount = Math.min(vmList.size(), size);
for (long i = 0; i < tobeScaledVMCount; i++) {
- KubernetesClusterVmMapVO vmMapVO = vmList.get((int) i);
- UserVmVO userVM = userVmDao.findById(vmMapVO.getVmId());
- boolean result = false;
- try {
- result = userVmManager.upgradeVirtualMachine(userVM.getId(), serviceOffering.getId(), new HashMap<String, String>());
- } catch (RuntimeException | ResourceUnavailableException | ManagementServerException | VirtualMachineMigrationException e) {
- logTransitStateAndThrow(Level.ERROR, String.format("Scaling Kubernetes cluster : %s failed, unable to scale cluster VM : %s due to %s", kubernetesCluster.getName(), userVM.getDisplayName(), e.getMessage()), kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed, e);
- }
- if (!result) {
- logTransitStateAndThrow(Level.WARN, String.format("Scaling Kubernetes cluster : %s failed, unable to scale cluster VM : %s", kubernetesCluster.getName(), userVM.getDisplayName()),kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
- }
+ new NodeOfferingResize(vmList.get((int) i), nodeType, serviceOffering).execute();
Review Comment:
I kept the single scale timeout, the same way the upgrade worker uses its timeout, so the setting keeps its meaning. To make it go further, the repeated cordon/uncordon polling is gone, and the time is now checked before each node starts instead of after it finishes, so a node that already succeeded is no longer reported as a timeout. Once the time is up, no further node is started and kubelet is not restarted. If you'd still prefer a separate per-node budget, I can add it as its own setting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6469461429
@andrijapanicsb a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "blueorangutan (via GitHub)" <gi...@apache.org>.
blueorangutan commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6471854816
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19802
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "andrijapanicsb (via GitHub) via commits" <co...@cloudstack.apache.org>.
andrijapanicsb commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6702537224
@ACSHomeBot package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (WIP)CKS: reconcile node capacity after live resize [cloudstack]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-5776719318
<!-- coverage-grade-bot -->
## 🔴 Test Coverage Grade: `D` — Marginal
| Metric | Value |
|--------|-------|
| Line coverage | **24.83%** |
| Branch coverage | **19.07%** |
### Grade Scale
| Grade | Line Coverage | Meaning |
|-------|--------------|---------|
| 🟢 A | ≥ 80% | Excellent - this code sleeps well at night 😴 |
| 🟡 B | 60-79% | Good - almost there, don't stop now 😉 |
| 🟠 C | 40-59% | Acceptable - your code is wearing a seatbelt, but no airbags 😬 |
| 🔴 D | 20-39% | Marginal - boldly shipping where no test has gone before 🖖 |
| ⛔ F | < 20% | Failing - tests? what tests? 🔥 |
> Branch coverage is shown as a secondary signal. Grade is determined by **line coverage**.
> [View full Actions run](https://github.com/apache/cloudstack/actions/runs/35722928121)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (WIP)CKS: reconcile node capacity after live resize [cloudstack]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-5779925802
## [](https://sonarcloud.io/dashboard?id=apache_cloudstack&pullRequest=14223) **Quality Gate failed**
Failed conditions
 [28.9% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_cloudstack&pullRequest=14223&metric=new_coverage&view=list) (required ≥ 40%)
<!-- sqra:qg_revision=b21a6ce495fd81d2cbc417e4fa9d19786fb3d21b19b321f0ccda97d4e462a9fc -->
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_cloudstack&pullRequest=14223)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (WIP)CKS: reconcile node capacity after live resize [cloudstack]
Posted by "andrijapanicsb (via GitHub)" <gi...@apache.org>.
andrijapanicsb commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6292091074
@ACSHomeBot package G
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (WIP)CKS: reconcile node capacity after live resize [cloudstack]
Posted by "ACSHomeBot (via GitHub)" <gi...@apache.org>.
ACSHomeBot commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6295851901
Package building started on **worker G**.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "Pearl1594 (via GitHub)" <gi...@apache.org>.
Pearl1594 commented on code in PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#discussion_r4384297973
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/utils/KubernetesClusterNodeCapacityReconcilerImpl.java:
##########
@@ -0,0 +1,232 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.kubernetes.cluster.utils;
+
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import com.cloud.kubernetes.cluster.KubernetesCluster;
+import com.cloud.kubernetes.cluster.KubernetesServiceHelper.KubernetesClusterNodeType;
+import com.cloud.offering.ServiceOffering;
+import com.cloud.uservm.UserVm;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.ssh.SshHelper;
+import com.google.gson.JsonArray;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParser;
+import org.apache.commons.lang3.StringUtils;
+
+/** SSH-backed implementation for a rolling CKS live-resize reconciliation. */
+public class KubernetesClusterNodeCapacityReconcilerImpl implements KubernetesClusterNodeCapacityReconciler {
+ // KVM QA observes 4,568,788 KiB in a 4,608 MiB guest after memory hotplug
+ // (about 146 MiB below the offering), so keep a small margin above that
+ // platform overhead while still rejecting materially undersized guests.
+ static final long MINIMUM_MEMORY_OVERHEAD_BYTES = 160L * 1024L * 1024L;
+ static final long KUBERNETES_MEMORY_REPORTING_TOLERANCE_BYTES = 16L * 1024L * 1024L;
+ private static final int COMMAND_TIMEOUT_MS = 30000;
+ private static final int POLL_INTERVAL_MS = 5000;
+ private static final Pattern NODE_NAME = Pattern.compile("[A-Za-z0-9][A-Za-z0-9.-]{0,252}");
+ private static final Pattern MEMORY_QUANTITY = Pattern.compile("([0-9]+)(Ki|Mi|Gi|Ti|K|M|G|T)?");
+ private static final String RESIZE_ANNOTATION = "cloudstack.apache.org/cks-live-resize";
+ static final String ONLINE_HOTPLUGGED_RESOURCES = "sudo sh -c 'for cpu in /sys/devices/system/cpu/cpu[0-9]*; do "
+ + "[ -e \"$cpu/online\" ] || continue; echo 1 > \"$cpu/online\"; done; "
+ + "for memory in /sys/devices/system/memory/memory[0-9]*; do "
+ + "[ -e \"$memory/online\" ] || continue; echo online > \"$memory/online\"; done'";
+
+ @Override
+ public boolean requiresKubeletRefresh(ServiceOffering oldOffering, ServiceOffering newOffering,
+ KubernetesClusterNodeType nodeType, KubernetesCluster.State originalState) {
+ return KubernetesCluster.State.Running == originalState
+ && (KubernetesClusterNodeType.WORKER == nodeType || KubernetesClusterNodeType.CONTROL == nodeType)
+ && capacityChanged(oldOffering, newOffering);
+ }
+
+ @Override
+ public NodeCapacitySnapshot captureBefore(KubernetesCluster cluster, UserVm vm, NodeAccess access) {
+ return nodeSnapshot(cluster, vm, access, guestResources(access));
+ }
+
+ @Override
+ public void cordonIfNeeded(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "=" + cluster.getUuid() + " --overwrite");
+ executeControl(access, "sudo /opt/bin/kubectl cordon " + node);
+ while (System.currentTimeMillis() < deadline) {
+ if (nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("CORDON", vm, "Kubernetes node did not become unschedulable");
+ }
+
+ @Override
+ public void onlineGuestResources(NodeAccess access) {
+ executeNode(access, ONLINE_HOTPLUGGED_RESOURCES);
+ }
+
+ @Override
+ public void verifyGuestResources(UserVm vm, ServiceOffering target, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ if (guestMatchesTarget(guestResources(access), target)) return;
+ sleep();
+ }
+ throw failure("GUEST_VERIFY", vm, "guest CPU or memory did not reach the target offering");
+ }
+
+ @Override
+ public boolean isKubernetesResourcesCurrent(NodeCapacitySnapshot snapshot, ServiceOffering target) {
+ GuestResources guest = new GuestResources(snapshot.getGuestOnlineCpuCount(), snapshot.getGuestMemoryKiB());
+ return guestMatchesTarget(guest, target) && kubernetesMatchesGuest(snapshot, snapshot);
+ }
+
+ @Override
+ public void restartKubelet(UserVm vm, NodeAccess access, long deadline) {
+ executeNode(access, "sudo systemctl restart kubelet");
+ while (System.currentTimeMillis() < deadline) {
+ if ("active".equals(executeNode(access, "sudo systemctl is-active kubelet").trim())) return;
+ sleep();
+ }
+ throw failure("KUBELET_RESTART", vm, "kubelet did not become active");
+ }
+
+ @Override
+ public NodeCapacitySnapshot waitForKubernetesResources(KubernetesCluster cluster, UserVm vm, ServiceOffering target,
+ NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ GuestResources guest = guestResources(access);
+ NodeCapacitySnapshot observed = nodeSnapshot(cluster, vm, access, guest);
+ if (guestMatchesTarget(guest, target) && kubernetesMatchesGuest(observed, before)) return observed;
+ sleep();
+ }
+ throw failure("CAPACITY_VERIFY", vm, "Kubernetes capacity did not match the resized guest");
+ }
+
+ @Override
+ public void restoreSchedulability(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl uncordon " + node);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "-");
+ while (System.currentTimeMillis() < deadline) {
+ if (!nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("RESTORE_SCHEDULABILITY", vm, "Kubernetes node remained unschedulable");
+ }
+
+ public static boolean capacityChanged(ServiceOffering oldOffering, ServiceOffering targetOffering) {
+ return oldOffering != null && targetOffering != null
+ && (!Objects.equals(oldOffering.getCpu(), targetOffering.getCpu())
+ || !Objects.equals(oldOffering.getRamSize(), targetOffering.getRamSize()));
+ }
+
+ static long parseCpuMillis(String value) { return value.endsWith("m") ? Long.parseLong(value.substring(0, value.length() - 1)) : Long.parseLong(value) * 1000L; }
+ static long parseMemoryBytes(String value) {
+ Matcher matcher = MEMORY_QUANTITY.matcher(value);
+ if (!matcher.matches()) throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ long number = Long.parseLong(matcher.group(1)); String unit = matcher.group(2);
+ if (unit == null) return number;
+ switch (unit) {
+ case "Ki": return number * 1024L; case "Mi": return number * 1024L * 1024L; case "Gi": return number * 1024L * 1024L * 1024L;
+ case "Ti": return number * 1024L * 1024L * 1024L * 1024L; case "K": return number * 1000L; case "M": return number * 1000L * 1000L;
+ case "G": return number * 1000L * 1000L * 1000L; case "T": return number * 1000L * 1000L * 1000L * 1000L;
+ default: throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ }
+ }
+
+ private NodeCapacitySnapshot nodeSnapshot(KubernetesCluster cluster, UserVm vm, NodeAccess access, GuestResources guest) {
+ JsonObject node = new JsonParser().parse(executeControl(access, "sudo /opt/bin/kubectl get node " + nodeName(vm) + " -o json")).getAsJsonObject();
+ JsonObject annotations = node.getAsJsonObject("metadata").has("annotations") ? node.getAsJsonObject("metadata").getAsJsonObject("annotations") : null;
+ boolean cksCordon = annotations != null && annotations.has(RESIZE_ANNOTATION) && cluster.getUuid().equals(annotations.get(RESIZE_ANNOTATION).getAsString());
+ JsonObject spec = node.has("spec") ? node.getAsJsonObject("spec") : new JsonObject(); JsonObject status = node.getAsJsonObject("status");
Review Comment:
new line for status - as sonar has pointed as well
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/utils/KubernetesClusterNodeCapacityReconcilerImpl.java:
##########
@@ -0,0 +1,232 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.kubernetes.cluster.utils;
+
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import com.cloud.kubernetes.cluster.KubernetesCluster;
+import com.cloud.kubernetes.cluster.KubernetesServiceHelper.KubernetesClusterNodeType;
+import com.cloud.offering.ServiceOffering;
+import com.cloud.uservm.UserVm;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.ssh.SshHelper;
+import com.google.gson.JsonArray;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParser;
+import org.apache.commons.lang3.StringUtils;
+
+/** SSH-backed implementation for a rolling CKS live-resize reconciliation. */
+public class KubernetesClusterNodeCapacityReconcilerImpl implements KubernetesClusterNodeCapacityReconciler {
+ // KVM QA observes 4,568,788 KiB in a 4,608 MiB guest after memory hotplug
+ // (about 146 MiB below the offering), so keep a small margin above that
+ // platform overhead while still rejecting materially undersized guests.
+ static final long MINIMUM_MEMORY_OVERHEAD_BYTES = 160L * 1024L * 1024L;
+ static final long KUBERNETES_MEMORY_REPORTING_TOLERANCE_BYTES = 16L * 1024L * 1024L;
+ private static final int COMMAND_TIMEOUT_MS = 30000;
+ private static final int POLL_INTERVAL_MS = 5000;
+ private static final Pattern NODE_NAME = Pattern.compile("[A-Za-z0-9][A-Za-z0-9.-]{0,252}");
+ private static final Pattern MEMORY_QUANTITY = Pattern.compile("([0-9]+)(Ki|Mi|Gi|Ti|K|M|G|T)?");
+ private static final String RESIZE_ANNOTATION = "cloudstack.apache.org/cks-live-resize";
+ static final String ONLINE_HOTPLUGGED_RESOURCES = "sudo sh -c 'for cpu in /sys/devices/system/cpu/cpu[0-9]*; do "
+ + "[ -e \"$cpu/online\" ] || continue; echo 1 > \"$cpu/online\"; done; "
+ + "for memory in /sys/devices/system/memory/memory[0-9]*; do "
+ + "[ -e \"$memory/online\" ] || continue; echo online > \"$memory/online\"; done'";
+
+ @Override
+ public boolean requiresKubeletRefresh(ServiceOffering oldOffering, ServiceOffering newOffering,
+ KubernetesClusterNodeType nodeType, KubernetesCluster.State originalState) {
+ return KubernetesCluster.State.Running == originalState
+ && (KubernetesClusterNodeType.WORKER == nodeType || KubernetesClusterNodeType.CONTROL == nodeType)
+ && capacityChanged(oldOffering, newOffering);
+ }
+
+ @Override
+ public NodeCapacitySnapshot captureBefore(KubernetesCluster cluster, UserVm vm, NodeAccess access) {
+ return nodeSnapshot(cluster, vm, access, guestResources(access));
+ }
+
+ @Override
+ public void cordonIfNeeded(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "=" + cluster.getUuid() + " --overwrite");
+ executeControl(access, "sudo /opt/bin/kubectl cordon " + node);
+ while (System.currentTimeMillis() < deadline) {
+ if (nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("CORDON", vm, "Kubernetes node did not become unschedulable");
+ }
+
+ @Override
+ public void onlineGuestResources(NodeAccess access) {
+ executeNode(access, ONLINE_HOTPLUGGED_RESOURCES);
+ }
+
+ @Override
+ public void verifyGuestResources(UserVm vm, ServiceOffering target, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ if (guestMatchesTarget(guestResources(access), target)) return;
+ sleep();
+ }
+ throw failure("GUEST_VERIFY", vm, "guest CPU or memory did not reach the target offering");
+ }
+
+ @Override
+ public boolean isKubernetesResourcesCurrent(NodeCapacitySnapshot snapshot, ServiceOffering target) {
+ GuestResources guest = new GuestResources(snapshot.getGuestOnlineCpuCount(), snapshot.getGuestMemoryKiB());
+ return guestMatchesTarget(guest, target) && kubernetesMatchesGuest(snapshot, snapshot);
+ }
+
+ @Override
+ public void restartKubelet(UserVm vm, NodeAccess access, long deadline) {
+ executeNode(access, "sudo systemctl restart kubelet");
+ while (System.currentTimeMillis() < deadline) {
+ if ("active".equals(executeNode(access, "sudo systemctl is-active kubelet").trim())) return;
+ sleep();
+ }
+ throw failure("KUBELET_RESTART", vm, "kubelet did not become active");
+ }
+
+ @Override
+ public NodeCapacitySnapshot waitForKubernetesResources(KubernetesCluster cluster, UserVm vm, ServiceOffering target,
+ NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ GuestResources guest = guestResources(access);
+ NodeCapacitySnapshot observed = nodeSnapshot(cluster, vm, access, guest);
+ if (guestMatchesTarget(guest, target) && kubernetesMatchesGuest(observed, before)) return observed;
+ sleep();
+ }
+ throw failure("CAPACITY_VERIFY", vm, "Kubernetes capacity did not match the resized guest");
+ }
+
+ @Override
+ public void restoreSchedulability(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl uncordon " + node);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "-");
+ while (System.currentTimeMillis() < deadline) {
+ if (!nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("RESTORE_SCHEDULABILITY", vm, "Kubernetes node remained unschedulable");
+ }
+
+ public static boolean capacityChanged(ServiceOffering oldOffering, ServiceOffering targetOffering) {
+ return oldOffering != null && targetOffering != null
+ && (!Objects.equals(oldOffering.getCpu(), targetOffering.getCpu())
+ || !Objects.equals(oldOffering.getRamSize(), targetOffering.getRamSize()));
+ }
+
+ static long parseCpuMillis(String value) { return value.endsWith("m") ? Long.parseLong(value.substring(0, value.length() - 1)) : Long.parseLong(value) * 1000L; }
+ static long parseMemoryBytes(String value) {
+ Matcher matcher = MEMORY_QUANTITY.matcher(value);
+ if (!matcher.matches()) throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ long number = Long.parseLong(matcher.group(1)); String unit = matcher.group(2);
+ if (unit == null) return number;
+ switch (unit) {
+ case "Ki": return number * 1024L; case "Mi": return number * 1024L * 1024L; case "Gi": return number * 1024L * 1024L * 1024L;
+ case "Ti": return number * 1024L * 1024L * 1024L * 1024L; case "K": return number * 1000L; case "M": return number * 1000L * 1000L;
+ case "G": return number * 1000L * 1000L * 1000L; case "T": return number * 1000L * 1000L * 1000L * 1000L;
+ default: throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ }
+ }
+
+ private NodeCapacitySnapshot nodeSnapshot(KubernetesCluster cluster, UserVm vm, NodeAccess access, GuestResources guest) {
+ JsonObject node = new JsonParser().parse(executeControl(access, "sudo /opt/bin/kubectl get node " + nodeName(vm) + " -o json")).getAsJsonObject();
+ JsonObject annotations = node.getAsJsonObject("metadata").has("annotations") ? node.getAsJsonObject("metadata").getAsJsonObject("annotations") : null;
+ boolean cksCordon = annotations != null && annotations.has(RESIZE_ANNOTATION) && cluster.getUuid().equals(annotations.get(RESIZE_ANNOTATION).getAsString());
+ JsonObject spec = node.has("spec") ? node.getAsJsonObject("spec") : new JsonObject(); JsonObject status = node.getAsJsonObject("status");
+ JsonObject capacity = status.getAsJsonObject("capacity"); JsonObject allocatable = status.getAsJsonObject("allocatable");
+ return new NodeCapacitySnapshot(spec.has("unschedulable") && spec.get("unschedulable").getAsBoolean(), cksCordon,
+ guest == null ? 0 : guest.cpu, guest == null ? 0 : guest.memoryKiB, parseCpuMillis(capacity.get("cpu").getAsString()),
+ parseMemoryBytes(capacity.get("memory").getAsString()), parseCpuMillis(allocatable.get("cpu").getAsString()),
+ parseMemoryBytes(allocatable.get("memory").getAsString()), isReady(status));
+ }
+
+ private boolean isReady(JsonObject status) {
+ JsonArray conditions = status.getAsJsonArray("conditions");
+ for (JsonElement condition : conditions) { JsonObject item = condition.getAsJsonObject(); if ("Ready".equals(item.get("type").getAsString()) && "True".equals(item.get("status").getAsString())) return true; }
Review Comment:
new lines - for better readability
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/utils/KubernetesClusterNodeCapacityReconcilerImpl.java:
##########
@@ -0,0 +1,232 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.kubernetes.cluster.utils;
+
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import com.cloud.kubernetes.cluster.KubernetesCluster;
+import com.cloud.kubernetes.cluster.KubernetesServiceHelper.KubernetesClusterNodeType;
+import com.cloud.offering.ServiceOffering;
+import com.cloud.uservm.UserVm;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.ssh.SshHelper;
+import com.google.gson.JsonArray;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParser;
+import org.apache.commons.lang3.StringUtils;
+
+/** SSH-backed implementation for a rolling CKS live-resize reconciliation. */
+public class KubernetesClusterNodeCapacityReconcilerImpl implements KubernetesClusterNodeCapacityReconciler {
+ // KVM QA observes 4,568,788 KiB in a 4,608 MiB guest after memory hotplug
+ // (about 146 MiB below the offering), so keep a small margin above that
+ // platform overhead while still rejecting materially undersized guests.
+ static final long MINIMUM_MEMORY_OVERHEAD_BYTES = 160L * 1024L * 1024L;
+ static final long KUBERNETES_MEMORY_REPORTING_TOLERANCE_BYTES = 16L * 1024L * 1024L;
+ private static final int COMMAND_TIMEOUT_MS = 30000;
+ private static final int POLL_INTERVAL_MS = 5000;
+ private static final Pattern NODE_NAME = Pattern.compile("[A-Za-z0-9][A-Za-z0-9.-]{0,252}");
+ private static final Pattern MEMORY_QUANTITY = Pattern.compile("([0-9]+)(Ki|Mi|Gi|Ti|K|M|G|T)?");
+ private static final String RESIZE_ANNOTATION = "cloudstack.apache.org/cks-live-resize";
+ static final String ONLINE_HOTPLUGGED_RESOURCES = "sudo sh -c 'for cpu in /sys/devices/system/cpu/cpu[0-9]*; do "
+ + "[ -e \"$cpu/online\" ] || continue; echo 1 > \"$cpu/online\"; done; "
+ + "for memory in /sys/devices/system/memory/memory[0-9]*; do "
+ + "[ -e \"$memory/online\" ] || continue; echo online > \"$memory/online\"; done'";
+
+ @Override
+ public boolean requiresKubeletRefresh(ServiceOffering oldOffering, ServiceOffering newOffering,
+ KubernetesClusterNodeType nodeType, KubernetesCluster.State originalState) {
+ return KubernetesCluster.State.Running == originalState
+ && (KubernetesClusterNodeType.WORKER == nodeType || KubernetesClusterNodeType.CONTROL == nodeType)
+ && capacityChanged(oldOffering, newOffering);
+ }
+
+ @Override
+ public NodeCapacitySnapshot captureBefore(KubernetesCluster cluster, UserVm vm, NodeAccess access) {
+ return nodeSnapshot(cluster, vm, access, guestResources(access));
+ }
+
+ @Override
+ public void cordonIfNeeded(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "=" + cluster.getUuid() + " --overwrite");
+ executeControl(access, "sudo /opt/bin/kubectl cordon " + node);
+ while (System.currentTimeMillis() < deadline) {
+ if (nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("CORDON", vm, "Kubernetes node did not become unschedulable");
+ }
+
+ @Override
+ public void onlineGuestResources(NodeAccess access) {
+ executeNode(access, ONLINE_HOTPLUGGED_RESOURCES);
+ }
+
+ @Override
+ public void verifyGuestResources(UserVm vm, ServiceOffering target, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ if (guestMatchesTarget(guestResources(access), target)) return;
+ sleep();
+ }
+ throw failure("GUEST_VERIFY", vm, "guest CPU or memory did not reach the target offering");
+ }
+
+ @Override
+ public boolean isKubernetesResourcesCurrent(NodeCapacitySnapshot snapshot, ServiceOffering target) {
+ GuestResources guest = new GuestResources(snapshot.getGuestOnlineCpuCount(), snapshot.getGuestMemoryKiB());
+ return guestMatchesTarget(guest, target) && kubernetesMatchesGuest(snapshot, snapshot);
+ }
+
+ @Override
+ public void restartKubelet(UserVm vm, NodeAccess access, long deadline) {
+ executeNode(access, "sudo systemctl restart kubelet");
+ while (System.currentTimeMillis() < deadline) {
+ if ("active".equals(executeNode(access, "sudo systemctl is-active kubelet").trim())) return;
+ sleep();
+ }
+ throw failure("KUBELET_RESTART", vm, "kubelet did not become active");
+ }
+
+ @Override
+ public NodeCapacitySnapshot waitForKubernetesResources(KubernetesCluster cluster, UserVm vm, ServiceOffering target,
+ NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ GuestResources guest = guestResources(access);
+ NodeCapacitySnapshot observed = nodeSnapshot(cluster, vm, access, guest);
+ if (guestMatchesTarget(guest, target) && kubernetesMatchesGuest(observed, before)) return observed;
+ sleep();
+ }
+ throw failure("CAPACITY_VERIFY", vm, "Kubernetes capacity did not match the resized guest");
+ }
+
+ @Override
+ public void restoreSchedulability(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl uncordon " + node);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "-");
+ while (System.currentTimeMillis() < deadline) {
+ if (!nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("RESTORE_SCHEDULABILITY", vm, "Kubernetes node remained unschedulable");
+ }
+
+ public static boolean capacityChanged(ServiceOffering oldOffering, ServiceOffering targetOffering) {
+ return oldOffering != null && targetOffering != null
+ && (!Objects.equals(oldOffering.getCpu(), targetOffering.getCpu())
+ || !Objects.equals(oldOffering.getRamSize(), targetOffering.getRamSize()));
+ }
+
+ static long parseCpuMillis(String value) { return value.endsWith("m") ? Long.parseLong(value.substring(0, value.length() - 1)) : Long.parseLong(value) * 1000L; }
+ static long parseMemoryBytes(String value) {
+ Matcher matcher = MEMORY_QUANTITY.matcher(value);
+ if (!matcher.matches()) throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ long number = Long.parseLong(matcher.group(1)); String unit = matcher.group(2);
+ if (unit == null) return number;
+ switch (unit) {
+ case "Ki": return number * 1024L; case "Mi": return number * 1024L * 1024L; case "Gi": return number * 1024L * 1024L * 1024L;
+ case "Ti": return number * 1024L * 1024L * 1024L * 1024L; case "K": return number * 1000L; case "M": return number * 1000L * 1000L;
+ case "G": return number * 1000L * 1000L * 1000L; case "T": return number * 1000L * 1000L * 1000L * 1000L;
+ default: throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
Review Comment:
can you add new lines - hard to read.
Also can we use KiB, MiB, GiB in ByteScaleUtils instead of using 1024*1024..
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/utils/KubernetesClusterNodeCapacityReconcilerImpl.java:
##########
@@ -0,0 +1,232 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.kubernetes.cluster.utils;
+
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import com.cloud.kubernetes.cluster.KubernetesCluster;
+import com.cloud.kubernetes.cluster.KubernetesServiceHelper.KubernetesClusterNodeType;
+import com.cloud.offering.ServiceOffering;
+import com.cloud.uservm.UserVm;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.ssh.SshHelper;
+import com.google.gson.JsonArray;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParser;
+import org.apache.commons.lang3.StringUtils;
+
+/** SSH-backed implementation for a rolling CKS live-resize reconciliation. */
+public class KubernetesClusterNodeCapacityReconcilerImpl implements KubernetesClusterNodeCapacityReconciler {
+ // KVM QA observes 4,568,788 KiB in a 4,608 MiB guest after memory hotplug
+ // (about 146 MiB below the offering), so keep a small margin above that
+ // platform overhead while still rejecting materially undersized guests.
+ static final long MINIMUM_MEMORY_OVERHEAD_BYTES = 160L * 1024L * 1024L;
+ static final long KUBERNETES_MEMORY_REPORTING_TOLERANCE_BYTES = 16L * 1024L * 1024L;
+ private static final int COMMAND_TIMEOUT_MS = 30000;
+ private static final int POLL_INTERVAL_MS = 5000;
+ private static final Pattern NODE_NAME = Pattern.compile("[A-Za-z0-9][A-Za-z0-9.-]{0,252}");
+ private static final Pattern MEMORY_QUANTITY = Pattern.compile("([0-9]+)(Ki|Mi|Gi|Ti|K|M|G|T)?");
+ private static final String RESIZE_ANNOTATION = "cloudstack.apache.org/cks-live-resize";
+ static final String ONLINE_HOTPLUGGED_RESOURCES = "sudo sh -c 'for cpu in /sys/devices/system/cpu/cpu[0-9]*; do "
+ + "[ -e \"$cpu/online\" ] || continue; echo 1 > \"$cpu/online\"; done; "
+ + "for memory in /sys/devices/system/memory/memory[0-9]*; do "
+ + "[ -e \"$memory/online\" ] || continue; echo online > \"$memory/online\"; done'";
+
+ @Override
+ public boolean requiresKubeletRefresh(ServiceOffering oldOffering, ServiceOffering newOffering,
+ KubernetesClusterNodeType nodeType, KubernetesCluster.State originalState) {
+ return KubernetesCluster.State.Running == originalState
+ && (KubernetesClusterNodeType.WORKER == nodeType || KubernetesClusterNodeType.CONTROL == nodeType)
+ && capacityChanged(oldOffering, newOffering);
+ }
+
+ @Override
+ public NodeCapacitySnapshot captureBefore(KubernetesCluster cluster, UserVm vm, NodeAccess access) {
+ return nodeSnapshot(cluster, vm, access, guestResources(access));
+ }
+
+ @Override
+ public void cordonIfNeeded(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "=" + cluster.getUuid() + " --overwrite");
+ executeControl(access, "sudo /opt/bin/kubectl cordon " + node);
+ while (System.currentTimeMillis() < deadline) {
+ if (nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("CORDON", vm, "Kubernetes node did not become unschedulable");
+ }
+
+ @Override
+ public void onlineGuestResources(NodeAccess access) {
+ executeNode(access, ONLINE_HOTPLUGGED_RESOURCES);
+ }
+
+ @Override
+ public void verifyGuestResources(UserVm vm, ServiceOffering target, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ if (guestMatchesTarget(guestResources(access), target)) return;
+ sleep();
+ }
+ throw failure("GUEST_VERIFY", vm, "guest CPU or memory did not reach the target offering");
+ }
+
+ @Override
+ public boolean isKubernetesResourcesCurrent(NodeCapacitySnapshot snapshot, ServiceOffering target) {
+ GuestResources guest = new GuestResources(snapshot.getGuestOnlineCpuCount(), snapshot.getGuestMemoryKiB());
+ return guestMatchesTarget(guest, target) && kubernetesMatchesGuest(snapshot, snapshot);
+ }
+
+ @Override
+ public void restartKubelet(UserVm vm, NodeAccess access, long deadline) {
+ executeNode(access, "sudo systemctl restart kubelet");
+ while (System.currentTimeMillis() < deadline) {
+ if ("active".equals(executeNode(access, "sudo systemctl is-active kubelet").trim())) return;
+ sleep();
+ }
+ throw failure("KUBELET_RESTART", vm, "kubelet did not become active");
+ }
+
+ @Override
+ public NodeCapacitySnapshot waitForKubernetesResources(KubernetesCluster cluster, UserVm vm, ServiceOffering target,
+ NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ GuestResources guest = guestResources(access);
+ NodeCapacitySnapshot observed = nodeSnapshot(cluster, vm, access, guest);
+ if (guestMatchesTarget(guest, target) && kubernetesMatchesGuest(observed, before)) return observed;
+ sleep();
+ }
+ throw failure("CAPACITY_VERIFY", vm, "Kubernetes capacity did not match the resized guest");
+ }
+
+ @Override
+ public void restoreSchedulability(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl uncordon " + node);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "-");
+ while (System.currentTimeMillis() < deadline) {
+ if (!nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("RESTORE_SCHEDULABILITY", vm, "Kubernetes node remained unschedulable");
+ }
+
+ public static boolean capacityChanged(ServiceOffering oldOffering, ServiceOffering targetOffering) {
+ return oldOffering != null && targetOffering != null
+ && (!Objects.equals(oldOffering.getCpu(), targetOffering.getCpu())
+ || !Objects.equals(oldOffering.getRamSize(), targetOffering.getRamSize()));
+ }
+
+ static long parseCpuMillis(String value) { return value.endsWith("m") ? Long.parseLong(value.substring(0, value.length() - 1)) : Long.parseLong(value) * 1000L; }
+ static long parseMemoryBytes(String value) {
+ Matcher matcher = MEMORY_QUANTITY.matcher(value);
+ if (!matcher.matches()) throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ long number = Long.parseLong(matcher.group(1)); String unit = matcher.group(2);
+ if (unit == null) return number;
+ switch (unit) {
+ case "Ki": return number * 1024L; case "Mi": return number * 1024L * 1024L; case "Gi": return number * 1024L * 1024L * 1024L;
+ case "Ti": return number * 1024L * 1024L * 1024L * 1024L; case "K": return number * 1000L; case "M": return number * 1000L * 1000L;
+ case "G": return number * 1000L * 1000L * 1000L; case "T": return number * 1000L * 1000L * 1000L * 1000L;
+ default: throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ }
+ }
+
+ private NodeCapacitySnapshot nodeSnapshot(KubernetesCluster cluster, UserVm vm, NodeAccess access, GuestResources guest) {
+ JsonObject node = new JsonParser().parse(executeControl(access, "sudo /opt/bin/kubectl get node " + nodeName(vm) + " -o json")).getAsJsonObject();
+ JsonObject annotations = node.getAsJsonObject("metadata").has("annotations") ? node.getAsJsonObject("metadata").getAsJsonObject("annotations") : null;
+ boolean cksCordon = annotations != null && annotations.has(RESIZE_ANNOTATION) && cluster.getUuid().equals(annotations.get(RESIZE_ANNOTATION).getAsString());
+ JsonObject spec = node.has("spec") ? node.getAsJsonObject("spec") : new JsonObject(); JsonObject status = node.getAsJsonObject("status");
+ JsonObject capacity = status.getAsJsonObject("capacity"); JsonObject allocatable = status.getAsJsonObject("allocatable");
+ return new NodeCapacitySnapshot(spec.has("unschedulable") && spec.get("unschedulable").getAsBoolean(), cksCordon,
+ guest == null ? 0 : guest.cpu, guest == null ? 0 : guest.memoryKiB, parseCpuMillis(capacity.get("cpu").getAsString()),
+ parseMemoryBytes(capacity.get("memory").getAsString()), parseCpuMillis(allocatable.get("cpu").getAsString()),
+ parseMemoryBytes(allocatable.get("memory").getAsString()), isReady(status));
+ }
+
+ private boolean isReady(JsonObject status) {
+ JsonArray conditions = status.getAsJsonArray("conditions");
+ for (JsonElement condition : conditions) { JsonObject item = condition.getAsJsonObject(); if ("Ready".equals(item.get("type").getAsString()) && "True".equals(item.get("status").getAsString())) return true; }
+ return false;
+ }
+
+ private GuestResources guestResources(NodeAccess access) {
+ String[] values = executeNode(access, "getconf _NPROCESSORS_ONLN; awk '/^MemTotal:/ {print $2}' /proc/meminfo").trim().split("\\s+");
+ if (values.length != 2) throw new CloudRuntimeException("Unable to read guest CPU and memory");
+ return new GuestResources(Long.parseLong(values[0]), Long.parseLong(values[1]));
+ }
+
+ private boolean guestMatchesTarget(GuestResources observed, ServiceOffering target) {
+ long targetMemory = target.getRamSize() * 1024L * 1024L; long observedMemory = observed.memoryKiB * 1024L;
+ long overhead = Math.max(targetMemory / 50L, MINIMUM_MEMORY_OVERHEAD_BYTES);
+ return observed.cpu == target.getCpu() && observedMemory <= targetMemory && targetMemory - observedMemory <= overhead;
+ }
+
+ private boolean kubernetesMatchesGuest(NodeCapacitySnapshot observed, NodeCapacitySnapshot before) {
+ return observed.isReady() && observed.getCapacityCpuMillis() == observed.getGuestOnlineCpuCount() * 1000L
+ && Math.abs(observed.getCapacityMemoryBytes() - observed.getGuestMemoryKiB() * 1024L) <= KUBERNETES_MEMORY_REPORTING_TOLERANCE_BYTES
+ && (observed.getGuestOnlineCpuCount() <= before.getGuestOnlineCpuCount() || observed.getAllocatableCpuMillis() > before.getAllocatableCpuMillis())
+ && (observed.getGuestMemoryKiB() <= before.getGuestMemoryKiB() || observed.getAllocatableMemoryBytes() > before.getAllocatableMemoryBytes());
+ }
+
+ protected String executeControl(NodeAccess access, String command) {
+ return execute(access.getControlAddress(), access.getControlPort(), access, command);
+ }
+
+ protected String executeNode(NodeAccess access, String command) {
+ return execute(access.getNodeAddress(), access.getNodePort(), access, command);
+ }
+
+ private String execute(String address, int port, NodeAccess access, String command) {
+ Pair<Boolean, String> result;
+ try {
+ result = SshHelper.sshExecute(address, port, access.getUser(), access.getSshKeyFile(), null, command, 10000, 10000, COMMAND_TIMEOUT_MS);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new NodeCapacityReconciliationException("CKS live-resize command interrupted", e);
+ } catch (RuntimeException e) {
+ throw e;
+ } catch (Exception e) {
+ // SshHelper declares Exception, including authentication failures.
+ // Translate only at this legacy boundary, not around the resize workflow.
+ throw new NodeCapacityReconciliationException("CKS live-resize SSH command failed", e);
+ }
+ if (!Boolean.TRUE.equals(result.first())) {
+ throw new NodeCapacityReconciliationException("CKS live-resize command failed");
+ }
+ return StringUtils.defaultString(result.second());
+ }
+ private String nodeName(UserVm vm) { String name = StringUtils.lowerCase(vm.getHostName()); if (!NODE_NAME.matcher(name).matches()) throw new CloudRuntimeException("Invalid Kubernetes node name for live resize"); return name; }
Review Comment:
new lines
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/utils/KubernetesClusterNodeCapacityReconcilerImpl.java:
##########
@@ -0,0 +1,232 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.kubernetes.cluster.utils;
+
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import com.cloud.kubernetes.cluster.KubernetesCluster;
+import com.cloud.kubernetes.cluster.KubernetesServiceHelper.KubernetesClusterNodeType;
+import com.cloud.offering.ServiceOffering;
+import com.cloud.uservm.UserVm;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.ssh.SshHelper;
+import com.google.gson.JsonArray;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParser;
+import org.apache.commons.lang3.StringUtils;
+
+/** SSH-backed implementation for a rolling CKS live-resize reconciliation. */
+public class KubernetesClusterNodeCapacityReconcilerImpl implements KubernetesClusterNodeCapacityReconciler {
+ // KVM QA observes 4,568,788 KiB in a 4,608 MiB guest after memory hotplug
+ // (about 146 MiB below the offering), so keep a small margin above that
+ // platform overhead while still rejecting materially undersized guests.
+ static final long MINIMUM_MEMORY_OVERHEAD_BYTES = 160L * 1024L * 1024L;
+ static final long KUBERNETES_MEMORY_REPORTING_TOLERANCE_BYTES = 16L * 1024L * 1024L;
+ private static final int COMMAND_TIMEOUT_MS = 30000;
+ private static final int POLL_INTERVAL_MS = 5000;
+ private static final Pattern NODE_NAME = Pattern.compile("[A-Za-z0-9][A-Za-z0-9.-]{0,252}");
+ private static final Pattern MEMORY_QUANTITY = Pattern.compile("([0-9]+)(Ki|Mi|Gi|Ti|K|M|G|T)?");
+ private static final String RESIZE_ANNOTATION = "cloudstack.apache.org/cks-live-resize";
+ static final String ONLINE_HOTPLUGGED_RESOURCES = "sudo sh -c 'for cpu in /sys/devices/system/cpu/cpu[0-9]*; do "
+ + "[ -e \"$cpu/online\" ] || continue; echo 1 > \"$cpu/online\"; done; "
+ + "for memory in /sys/devices/system/memory/memory[0-9]*; do "
+ + "[ -e \"$memory/online\" ] || continue; echo online > \"$memory/online\"; done'";
+
+ @Override
+ public boolean requiresKubeletRefresh(ServiceOffering oldOffering, ServiceOffering newOffering,
+ KubernetesClusterNodeType nodeType, KubernetesCluster.State originalState) {
+ return KubernetesCluster.State.Running == originalState
+ && (KubernetesClusterNodeType.WORKER == nodeType || KubernetesClusterNodeType.CONTROL == nodeType)
+ && capacityChanged(oldOffering, newOffering);
+ }
+
+ @Override
+ public NodeCapacitySnapshot captureBefore(KubernetesCluster cluster, UserVm vm, NodeAccess access) {
+ return nodeSnapshot(cluster, vm, access, guestResources(access));
+ }
+
+ @Override
+ public void cordonIfNeeded(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "=" + cluster.getUuid() + " --overwrite");
+ executeControl(access, "sudo /opt/bin/kubectl cordon " + node);
+ while (System.currentTimeMillis() < deadline) {
+ if (nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("CORDON", vm, "Kubernetes node did not become unschedulable");
+ }
+
+ @Override
+ public void onlineGuestResources(NodeAccess access) {
+ executeNode(access, ONLINE_HOTPLUGGED_RESOURCES);
+ }
+
+ @Override
+ public void verifyGuestResources(UserVm vm, ServiceOffering target, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ if (guestMatchesTarget(guestResources(access), target)) return;
+ sleep();
+ }
+ throw failure("GUEST_VERIFY", vm, "guest CPU or memory did not reach the target offering");
+ }
+
+ @Override
+ public boolean isKubernetesResourcesCurrent(NodeCapacitySnapshot snapshot, ServiceOffering target) {
+ GuestResources guest = new GuestResources(snapshot.getGuestOnlineCpuCount(), snapshot.getGuestMemoryKiB());
+ return guestMatchesTarget(guest, target) && kubernetesMatchesGuest(snapshot, snapshot);
+ }
+
+ @Override
+ public void restartKubelet(UserVm vm, NodeAccess access, long deadline) {
+ executeNode(access, "sudo systemctl restart kubelet");
+ while (System.currentTimeMillis() < deadline) {
+ if ("active".equals(executeNode(access, "sudo systemctl is-active kubelet").trim())) return;
+ sleep();
+ }
+ throw failure("KUBELET_RESTART", vm, "kubelet did not become active");
+ }
+
+ @Override
+ public NodeCapacitySnapshot waitForKubernetesResources(KubernetesCluster cluster, UserVm vm, ServiceOffering target,
+ NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ GuestResources guest = guestResources(access);
+ NodeCapacitySnapshot observed = nodeSnapshot(cluster, vm, access, guest);
+ if (guestMatchesTarget(guest, target) && kubernetesMatchesGuest(observed, before)) return observed;
+ sleep();
+ }
+ throw failure("CAPACITY_VERIFY", vm, "Kubernetes capacity did not match the resized guest");
+ }
+
+ @Override
+ public void restoreSchedulability(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl uncordon " + node);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "-");
+ while (System.currentTimeMillis() < deadline) {
+ if (!nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("RESTORE_SCHEDULABILITY", vm, "Kubernetes node remained unschedulable");
+ }
+
+ public static boolean capacityChanged(ServiceOffering oldOffering, ServiceOffering targetOffering) {
+ return oldOffering != null && targetOffering != null
+ && (!Objects.equals(oldOffering.getCpu(), targetOffering.getCpu())
+ || !Objects.equals(oldOffering.getRamSize(), targetOffering.getRamSize()));
+ }
+
+ static long parseCpuMillis(String value) { return value.endsWith("m") ? Long.parseLong(value.substring(0, value.length() - 1)) : Long.parseLong(value) * 1000L; }
+ static long parseMemoryBytes(String value) {
+ Matcher matcher = MEMORY_QUANTITY.matcher(value);
+ if (!matcher.matches()) throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ long number = Long.parseLong(matcher.group(1)); String unit = matcher.group(2);
+ if (unit == null) return number;
+ switch (unit) {
+ case "Ki": return number * 1024L; case "Mi": return number * 1024L * 1024L; case "Gi": return number * 1024L * 1024L * 1024L;
+ case "Ti": return number * 1024L * 1024L * 1024L * 1024L; case "K": return number * 1000L; case "M": return number * 1000L * 1000L;
+ case "G": return number * 1000L * 1000L * 1000L; case "T": return number * 1000L * 1000L * 1000L * 1000L;
+ default: throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ }
+ }
+
+ private NodeCapacitySnapshot nodeSnapshot(KubernetesCluster cluster, UserVm vm, NodeAccess access, GuestResources guest) {
+ JsonObject node = new JsonParser().parse(executeControl(access, "sudo /opt/bin/kubectl get node " + nodeName(vm) + " -o json")).getAsJsonObject();
+ JsonObject annotations = node.getAsJsonObject("metadata").has("annotations") ? node.getAsJsonObject("metadata").getAsJsonObject("annotations") : null;
+ boolean cksCordon = annotations != null && annotations.has(RESIZE_ANNOTATION) && cluster.getUuid().equals(annotations.get(RESIZE_ANNOTATION).getAsString());
+ JsonObject spec = node.has("spec") ? node.getAsJsonObject("spec") : new JsonObject(); JsonObject status = node.getAsJsonObject("status");
+ JsonObject capacity = status.getAsJsonObject("capacity"); JsonObject allocatable = status.getAsJsonObject("allocatable");
+ return new NodeCapacitySnapshot(spec.has("unschedulable") && spec.get("unschedulable").getAsBoolean(), cksCordon,
+ guest == null ? 0 : guest.cpu, guest == null ? 0 : guest.memoryKiB, parseCpuMillis(capacity.get("cpu").getAsString()),
+ parseMemoryBytes(capacity.get("memory").getAsString()), parseCpuMillis(allocatable.get("cpu").getAsString()),
+ parseMemoryBytes(allocatable.get("memory").getAsString()), isReady(status));
+ }
+
+ private boolean isReady(JsonObject status) {
+ JsonArray conditions = status.getAsJsonArray("conditions");
+ for (JsonElement condition : conditions) { JsonObject item = condition.getAsJsonObject(); if ("Ready".equals(item.get("type").getAsString()) && "True".equals(item.get("status").getAsString())) return true; }
+ return false;
+ }
+
+ private GuestResources guestResources(NodeAccess access) {
+ String[] values = executeNode(access, "getconf _NPROCESSORS_ONLN; awk '/^MemTotal:/ {print $2}' /proc/meminfo").trim().split("\\s+");
+ if (values.length != 2) throw new CloudRuntimeException("Unable to read guest CPU and memory");
Review Comment:
The general style in this codebase is to use braces even for a single-line if body. For uniformity, could you please update these if statements to use braces:
```suggestion
if (values.length != 2) {
throw new CloudRuntimeException("Unable to read guest CPU and memory");
}
```
There are other places too where this is missed - can you please address it there as well.
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/utils/KubernetesClusterNodeCapacityReconcilerImpl.java:
##########
@@ -0,0 +1,232 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.kubernetes.cluster.utils;
+
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import com.cloud.kubernetes.cluster.KubernetesCluster;
+import com.cloud.kubernetes.cluster.KubernetesServiceHelper.KubernetesClusterNodeType;
+import com.cloud.offering.ServiceOffering;
+import com.cloud.uservm.UserVm;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.ssh.SshHelper;
+import com.google.gson.JsonArray;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParser;
+import org.apache.commons.lang3.StringUtils;
+
+/** SSH-backed implementation for a rolling CKS live-resize reconciliation. */
+public class KubernetesClusterNodeCapacityReconcilerImpl implements KubernetesClusterNodeCapacityReconciler {
+ // KVM QA observes 4,568,788 KiB in a 4,608 MiB guest after memory hotplug
+ // (about 146 MiB below the offering), so keep a small margin above that
+ // platform overhead while still rejecting materially undersized guests.
+ static final long MINIMUM_MEMORY_OVERHEAD_BYTES = 160L * 1024L * 1024L;
+ static final long KUBERNETES_MEMORY_REPORTING_TOLERANCE_BYTES = 16L * 1024L * 1024L;
+ private static final int COMMAND_TIMEOUT_MS = 30000;
+ private static final int POLL_INTERVAL_MS = 5000;
+ private static final Pattern NODE_NAME = Pattern.compile("[A-Za-z0-9][A-Za-z0-9.-]{0,252}");
+ private static final Pattern MEMORY_QUANTITY = Pattern.compile("([0-9]+)(Ki|Mi|Gi|Ti|K|M|G|T)?");
+ private static final String RESIZE_ANNOTATION = "cloudstack.apache.org/cks-live-resize";
+ static final String ONLINE_HOTPLUGGED_RESOURCES = "sudo sh -c 'for cpu in /sys/devices/system/cpu/cpu[0-9]*; do "
+ + "[ -e \"$cpu/online\" ] || continue; echo 1 > \"$cpu/online\"; done; "
+ + "for memory in /sys/devices/system/memory/memory[0-9]*; do "
+ + "[ -e \"$memory/online\" ] || continue; echo online > \"$memory/online\"; done'";
+
+ @Override
+ public boolean requiresKubeletRefresh(ServiceOffering oldOffering, ServiceOffering newOffering,
+ KubernetesClusterNodeType nodeType, KubernetesCluster.State originalState) {
+ return KubernetesCluster.State.Running == originalState
+ && (KubernetesClusterNodeType.WORKER == nodeType || KubernetesClusterNodeType.CONTROL == nodeType)
+ && capacityChanged(oldOffering, newOffering);
+ }
+
+ @Override
+ public NodeCapacitySnapshot captureBefore(KubernetesCluster cluster, UserVm vm, NodeAccess access) {
+ return nodeSnapshot(cluster, vm, access, guestResources(access));
+ }
+
+ @Override
+ public void cordonIfNeeded(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "=" + cluster.getUuid() + " --overwrite");
+ executeControl(access, "sudo /opt/bin/kubectl cordon " + node);
+ while (System.currentTimeMillis() < deadline) {
+ if (nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("CORDON", vm, "Kubernetes node did not become unschedulable");
+ }
+
+ @Override
+ public void onlineGuestResources(NodeAccess access) {
+ executeNode(access, ONLINE_HOTPLUGGED_RESOURCES);
+ }
+
+ @Override
+ public void verifyGuestResources(UserVm vm, ServiceOffering target, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ if (guestMatchesTarget(guestResources(access), target)) return;
+ sleep();
+ }
+ throw failure("GUEST_VERIFY", vm, "guest CPU or memory did not reach the target offering");
+ }
+
+ @Override
+ public boolean isKubernetesResourcesCurrent(NodeCapacitySnapshot snapshot, ServiceOffering target) {
+ GuestResources guest = new GuestResources(snapshot.getGuestOnlineCpuCount(), snapshot.getGuestMemoryKiB());
+ return guestMatchesTarget(guest, target) && kubernetesMatchesGuest(snapshot, snapshot);
+ }
+
+ @Override
+ public void restartKubelet(UserVm vm, NodeAccess access, long deadline) {
+ executeNode(access, "sudo systemctl restart kubelet");
+ while (System.currentTimeMillis() < deadline) {
+ if ("active".equals(executeNode(access, "sudo systemctl is-active kubelet").trim())) return;
+ sleep();
+ }
+ throw failure("KUBELET_RESTART", vm, "kubelet did not become active");
+ }
+
+ @Override
+ public NodeCapacitySnapshot waitForKubernetesResources(KubernetesCluster cluster, UserVm vm, ServiceOffering target,
+ NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ GuestResources guest = guestResources(access);
+ NodeCapacitySnapshot observed = nodeSnapshot(cluster, vm, access, guest);
+ if (guestMatchesTarget(guest, target) && kubernetesMatchesGuest(observed, before)) return observed;
+ sleep();
+ }
+ throw failure("CAPACITY_VERIFY", vm, "Kubernetes capacity did not match the resized guest");
+ }
+
+ @Override
+ public void restoreSchedulability(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl uncordon " + node);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "-");
+ while (System.currentTimeMillis() < deadline) {
+ if (!nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("RESTORE_SCHEDULABILITY", vm, "Kubernetes node remained unschedulable");
+ }
+
+ public static boolean capacityChanged(ServiceOffering oldOffering, ServiceOffering targetOffering) {
+ return oldOffering != null && targetOffering != null
+ && (!Objects.equals(oldOffering.getCpu(), targetOffering.getCpu())
+ || !Objects.equals(oldOffering.getRamSize(), targetOffering.getRamSize()));
+ }
+
+ static long parseCpuMillis(String value) { return value.endsWith("m") ? Long.parseLong(value.substring(0, value.length() - 1)) : Long.parseLong(value) * 1000L; }
+ static long parseMemoryBytes(String value) {
+ Matcher matcher = MEMORY_QUANTITY.matcher(value);
+ if (!matcher.matches()) throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ long number = Long.parseLong(matcher.group(1)); String unit = matcher.group(2);
+ if (unit == null) return number;
+ switch (unit) {
+ case "Ki": return number * 1024L; case "Mi": return number * 1024L * 1024L; case "Gi": return number * 1024L * 1024L * 1024L;
+ case "Ti": return number * 1024L * 1024L * 1024L * 1024L; case "K": return number * 1000L; case "M": return number * 1000L * 1000L;
+ case "G": return number * 1000L * 1000L * 1000L; case "T": return number * 1000L * 1000L * 1000L * 1000L;
+ default: throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ }
+ }
+
+ private NodeCapacitySnapshot nodeSnapshot(KubernetesCluster cluster, UserVm vm, NodeAccess access, GuestResources guest) {
+ JsonObject node = new JsonParser().parse(executeControl(access, "sudo /opt/bin/kubectl get node " + nodeName(vm) + " -o json")).getAsJsonObject();
+ JsonObject annotations = node.getAsJsonObject("metadata").has("annotations") ? node.getAsJsonObject("metadata").getAsJsonObject("annotations") : null;
+ boolean cksCordon = annotations != null && annotations.has(RESIZE_ANNOTATION) && cluster.getUuid().equals(annotations.get(RESIZE_ANNOTATION).getAsString());
+ JsonObject spec = node.has("spec") ? node.getAsJsonObject("spec") : new JsonObject(); JsonObject status = node.getAsJsonObject("status");
+ JsonObject capacity = status.getAsJsonObject("capacity"); JsonObject allocatable = status.getAsJsonObject("allocatable");
+ return new NodeCapacitySnapshot(spec.has("unschedulable") && spec.get("unschedulable").getAsBoolean(), cksCordon,
+ guest == null ? 0 : guest.cpu, guest == null ? 0 : guest.memoryKiB, parseCpuMillis(capacity.get("cpu").getAsString()),
+ parseMemoryBytes(capacity.get("memory").getAsString()), parseCpuMillis(allocatable.get("cpu").getAsString()),
+ parseMemoryBytes(allocatable.get("memory").getAsString()), isReady(status));
+ }
+
+ private boolean isReady(JsonObject status) {
+ JsonArray conditions = status.getAsJsonArray("conditions");
+ for (JsonElement condition : conditions) { JsonObject item = condition.getAsJsonObject(); if ("Ready".equals(item.get("type").getAsString()) && "True".equals(item.get("status").getAsString())) return true; }
+ return false;
+ }
+
+ private GuestResources guestResources(NodeAccess access) {
+ String[] values = executeNode(access, "getconf _NPROCESSORS_ONLN; awk '/^MemTotal:/ {print $2}' /proc/meminfo").trim().split("\\s+");
+ if (values.length != 2) throw new CloudRuntimeException("Unable to read guest CPU and memory");
+ return new GuestResources(Long.parseLong(values[0]), Long.parseLong(values[1]));
+ }
+
+ private boolean guestMatchesTarget(GuestResources observed, ServiceOffering target) {
+ long targetMemory = target.getRamSize() * 1024L * 1024L; long observedMemory = observed.memoryKiB * 1024L;
+ long overhead = Math.max(targetMemory / 50L, MINIMUM_MEMORY_OVERHEAD_BYTES);
+ return observed.cpu == target.getCpu() && observedMemory <= targetMemory && targetMemory - observedMemory <= overhead;
+ }
+
+ private boolean kubernetesMatchesGuest(NodeCapacitySnapshot observed, NodeCapacitySnapshot before) {
+ return observed.isReady() && observed.getCapacityCpuMillis() == observed.getGuestOnlineCpuCount() * 1000L
+ && Math.abs(observed.getCapacityMemoryBytes() - observed.getGuestMemoryKiB() * 1024L) <= KUBERNETES_MEMORY_REPORTING_TOLERANCE_BYTES
+ && (observed.getGuestOnlineCpuCount() <= before.getGuestOnlineCpuCount() || observed.getAllocatableCpuMillis() > before.getAllocatableCpuMillis())
+ && (observed.getGuestMemoryKiB() <= before.getGuestMemoryKiB() || observed.getAllocatableMemoryBytes() > before.getAllocatableMemoryBytes());
+ }
+
+ protected String executeControl(NodeAccess access, String command) {
+ return execute(access.getControlAddress(), access.getControlPort(), access, command);
+ }
+
+ protected String executeNode(NodeAccess access, String command) {
+ return execute(access.getNodeAddress(), access.getNodePort(), access, command);
+ }
+
+ private String execute(String address, int port, NodeAccess access, String command) {
+ Pair<Boolean, String> result;
+ try {
+ result = SshHelper.sshExecute(address, port, access.getUser(), access.getSshKeyFile(), null, command, 10000, 10000, COMMAND_TIMEOUT_MS);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new NodeCapacityReconciliationException("CKS live-resize command interrupted", e);
+ } catch (RuntimeException e) {
+ throw e;
+ } catch (Exception e) {
+ // SshHelper declares Exception, including authentication failures.
+ // Translate only at this legacy boundary, not around the resize workflow.
+ throw new NodeCapacityReconciliationException("CKS live-resize SSH command failed", e);
+ }
+ if (!Boolean.TRUE.equals(result.first())) {
+ throw new NodeCapacityReconciliationException("CKS live-resize command failed");
+ }
+ return StringUtils.defaultString(result.second());
+ }
+ private String nodeName(UserVm vm) { String name = StringUtils.lowerCase(vm.getHostName()); if (!NODE_NAME.matcher(name).matches()) throw new CloudRuntimeException("Invalid Kubernetes node name for live resize"); return name; }
+ private NodeCapacityReconciliationException failure(String phase, UserVm vm, String message) {
+ return new NodeCapacityReconciliationException("CKS live resize " + phase + " failed for VM " + vm.getUuid() + ": " + message);
+ }
+
+ private void sleep() {
+ try {
+ Thread.sleep(POLL_INTERVAL_MS);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new NodeCapacityReconciliationException("CKS live-resize resource polling interrupted", e);
+ }
+ }
+ private static class GuestResources { private final long cpu; private final long memoryKiB; GuestResources(long cpu, long memoryKiB) { this.cpu = cpu; this.memoryKiB = memoryKiB; } }
Review Comment:
same here
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterScaleWorker.java:
##########
@@ -229,32 +236,36 @@ private KubernetesClusterVO updateKubernetesClusterEntryForNodeType(final Long n
}
protected Pair<Long, Long> calculateNewClusterCountAndCapacity(Long newWorkerSize, KubernetesClusterNodeType nodeType, ServiceOffering serviceOffering) {
- long cores;
- long memory;
- long totalClusterSize = newWorkerSize == null ? kubernetesCluster.getTotalNodeCount() : (newWorkerSize + kubernetesCluster.getControlNodeCount() + kubernetesCluster.getEtcdNodeCount());
-
- if (nodeType == DEFAULT) {
- cores = serviceOffering.getCpu() * totalClusterSize;
- memory = serviceOffering.getRamSize() * totalClusterSize;
- } else {
- long nodeCount = getNodeCountForType(nodeType, kubernetesCluster);
- Long existingOfferingId = getExistingOfferingIdForNodeType(nodeType, kubernetesCluster);
- if (existingOfferingId == null) {
- existingOfferingId = serviceOffering.getId();
- }
- ServiceOfferingVO previousOffering = serviceOfferingDao.findById(existingOfferingId);
- Pair<Long, Long> previousNodesCapacity = calculateNodesCapacity(previousOffering, nodeCount);
- if (WORKER == nodeType) {
- nodeCount = newWorkerSize == null ? kubernetesCluster.getNodeCount() : newWorkerSize;
+ long cores = 0;
+ long memory = 0;
+ // VM offerings have already changed here. Recompute the total from the
+ // configured node types instead of subtracting a supposedly old VM offering.
+ for (KubernetesClusterNodeType type : List.of(CONTROL, WORKER, ETCD)) {
Review Comment:
We already have a list defined here: `KubernetesClusterActionWorker.CLUSTER_NODES_TYPES_LIST`; can we reuse that?
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterScaleWorker.java:
##########
@@ -605,6 +753,9 @@ public boolean scaleCluster() throws CloudRuntimeException {
for (KubernetesClusterNodeType nodeType : Arrays.asList(CONTROL, ETCD, WORKER)) {
boolean isWorkerNode = WORKER == nodeType;
final long newVMRequired = (!isWorkerNode || clusterSize == null) ? 0 : clusterSize - originalClusterSize;
+ if (nodeType == ETCD && kubernetesCluster.getEtcdNodeCount() == 0) {
+ continue;
+ }
Review Comment:
This change doesn't seem to concern the live resize problem. #14138 already handles clusters with no offering for a node type that has no nodes, so we can remove it from this PR.
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/utils/KubernetesClusterNodeCapacityReconcilerImpl.java:
##########
@@ -0,0 +1,232 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.kubernetes.cluster.utils;
+
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import com.cloud.kubernetes.cluster.KubernetesCluster;
+import com.cloud.kubernetes.cluster.KubernetesServiceHelper.KubernetesClusterNodeType;
+import com.cloud.offering.ServiceOffering;
+import com.cloud.uservm.UserVm;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.ssh.SshHelper;
+import com.google.gson.JsonArray;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParser;
+import org.apache.commons.lang3.StringUtils;
+
+/** SSH-backed implementation for a rolling CKS live-resize reconciliation. */
+public class KubernetesClusterNodeCapacityReconcilerImpl implements KubernetesClusterNodeCapacityReconciler {
+ // KVM QA observes 4,568,788 KiB in a 4,608 MiB guest after memory hotplug
+ // (about 146 MiB below the offering), so keep a small margin above that
+ // platform overhead while still rejecting materially undersized guests.
+ static final long MINIMUM_MEMORY_OVERHEAD_BYTES = 160L * 1024L * 1024L;
+ static final long KUBERNETES_MEMORY_REPORTING_TOLERANCE_BYTES = 16L * 1024L * 1024L;
+ private static final int COMMAND_TIMEOUT_MS = 30000;
+ private static final int POLL_INTERVAL_MS = 5000;
+ private static final Pattern NODE_NAME = Pattern.compile("[A-Za-z0-9][A-Za-z0-9.-]{0,252}");
+ private static final Pattern MEMORY_QUANTITY = Pattern.compile("([0-9]+)(Ki|Mi|Gi|Ti|K|M|G|T)?");
+ private static final String RESIZE_ANNOTATION = "cloudstack.apache.org/cks-live-resize";
+ static final String ONLINE_HOTPLUGGED_RESOURCES = "sudo sh -c 'for cpu in /sys/devices/system/cpu/cpu[0-9]*; do "
+ + "[ -e \"$cpu/online\" ] || continue; echo 1 > \"$cpu/online\"; done; "
+ + "for memory in /sys/devices/system/memory/memory[0-9]*; do "
+ + "[ -e \"$memory/online\" ] || continue; echo online > \"$memory/online\"; done'";
+
+ @Override
+ public boolean requiresKubeletRefresh(ServiceOffering oldOffering, ServiceOffering newOffering,
+ KubernetesClusterNodeType nodeType, KubernetesCluster.State originalState) {
+ return KubernetesCluster.State.Running == originalState
+ && (KubernetesClusterNodeType.WORKER == nodeType || KubernetesClusterNodeType.CONTROL == nodeType)
+ && capacityChanged(oldOffering, newOffering);
+ }
+
+ @Override
+ public NodeCapacitySnapshot captureBefore(KubernetesCluster cluster, UserVm vm, NodeAccess access) {
+ return nodeSnapshot(cluster, vm, access, guestResources(access));
+ }
+
+ @Override
+ public void cordonIfNeeded(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "=" + cluster.getUuid() + " --overwrite");
+ executeControl(access, "sudo /opt/bin/kubectl cordon " + node);
+ while (System.currentTimeMillis() < deadline) {
+ if (nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("CORDON", vm, "Kubernetes node did not become unschedulable");
+ }
+
+ @Override
+ public void onlineGuestResources(NodeAccess access) {
+ executeNode(access, ONLINE_HOTPLUGGED_RESOURCES);
+ }
+
+ @Override
+ public void verifyGuestResources(UserVm vm, ServiceOffering target, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ if (guestMatchesTarget(guestResources(access), target)) return;
+ sleep();
+ }
+ throw failure("GUEST_VERIFY", vm, "guest CPU or memory did not reach the target offering");
+ }
+
+ @Override
+ public boolean isKubernetesResourcesCurrent(NodeCapacitySnapshot snapshot, ServiceOffering target) {
+ GuestResources guest = new GuestResources(snapshot.getGuestOnlineCpuCount(), snapshot.getGuestMemoryKiB());
+ return guestMatchesTarget(guest, target) && kubernetesMatchesGuest(snapshot, snapshot);
+ }
+
+ @Override
+ public void restartKubelet(UserVm vm, NodeAccess access, long deadline) {
+ executeNode(access, "sudo systemctl restart kubelet");
+ while (System.currentTimeMillis() < deadline) {
+ if ("active".equals(executeNode(access, "sudo systemctl is-active kubelet").trim())) return;
+ sleep();
+ }
+ throw failure("KUBELET_RESTART", vm, "kubelet did not become active");
+ }
+
+ @Override
+ public NodeCapacitySnapshot waitForKubernetesResources(KubernetesCluster cluster, UserVm vm, ServiceOffering target,
+ NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ GuestResources guest = guestResources(access);
+ NodeCapacitySnapshot observed = nodeSnapshot(cluster, vm, access, guest);
+ if (guestMatchesTarget(guest, target) && kubernetesMatchesGuest(observed, before)) return observed;
+ sleep();
+ }
+ throw failure("CAPACITY_VERIFY", vm, "Kubernetes capacity did not match the resized guest");
+ }
+
+ @Override
+ public void restoreSchedulability(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl uncordon " + node);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "-");
+ while (System.currentTimeMillis() < deadline) {
+ if (!nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("RESTORE_SCHEDULABILITY", vm, "Kubernetes node remained unschedulable");
+ }
+
+ public static boolean capacityChanged(ServiceOffering oldOffering, ServiceOffering targetOffering) {
+ return oldOffering != null && targetOffering != null
+ && (!Objects.equals(oldOffering.getCpu(), targetOffering.getCpu())
+ || !Objects.equals(oldOffering.getRamSize(), targetOffering.getRamSize()));
+ }
+
+ static long parseCpuMillis(String value) { return value.endsWith("m") ? Long.parseLong(value.substring(0, value.length() - 1)) : Long.parseLong(value) * 1000L; }
+ static long parseMemoryBytes(String value) {
+ Matcher matcher = MEMORY_QUANTITY.matcher(value);
+ if (!matcher.matches()) throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ long number = Long.parseLong(matcher.group(1)); String unit = matcher.group(2);
+ if (unit == null) return number;
+ switch (unit) {
+ case "Ki": return number * 1024L; case "Mi": return number * 1024L * 1024L; case "Gi": return number * 1024L * 1024L * 1024L;
+ case "Ti": return number * 1024L * 1024L * 1024L * 1024L; case "K": return number * 1000L; case "M": return number * 1000L * 1000L;
+ case "G": return number * 1000L * 1000L * 1000L; case "T": return number * 1000L * 1000L * 1000L * 1000L;
+ default: throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ }
+ }
+
+ private NodeCapacitySnapshot nodeSnapshot(KubernetesCluster cluster, UserVm vm, NodeAccess access, GuestResources guest) {
+ JsonObject node = new JsonParser().parse(executeControl(access, "sudo /opt/bin/kubectl get node " + nodeName(vm) + " -o json")).getAsJsonObject();
+ JsonObject annotations = node.getAsJsonObject("metadata").has("annotations") ? node.getAsJsonObject("metadata").getAsJsonObject("annotations") : null;
+ boolean cksCordon = annotations != null && annotations.has(RESIZE_ANNOTATION) && cluster.getUuid().equals(annotations.get(RESIZE_ANNOTATION).getAsString());
+ JsonObject spec = node.has("spec") ? node.getAsJsonObject("spec") : new JsonObject(); JsonObject status = node.getAsJsonObject("status");
+ JsonObject capacity = status.getAsJsonObject("capacity"); JsonObject allocatable = status.getAsJsonObject("allocatable");
+ return new NodeCapacitySnapshot(spec.has("unschedulable") && spec.get("unschedulable").getAsBoolean(), cksCordon,
+ guest == null ? 0 : guest.cpu, guest == null ? 0 : guest.memoryKiB, parseCpuMillis(capacity.get("cpu").getAsString()),
+ parseMemoryBytes(capacity.get("memory").getAsString()), parseCpuMillis(allocatable.get("cpu").getAsString()),
+ parseMemoryBytes(allocatable.get("memory").getAsString()), isReady(status));
+ }
+
+ private boolean isReady(JsonObject status) {
+ JsonArray conditions = status.getAsJsonArray("conditions");
+ for (JsonElement condition : conditions) { JsonObject item = condition.getAsJsonObject(); if ("Ready".equals(item.get("type").getAsString()) && "True".equals(item.get("status").getAsString())) return true; }
+ return false;
+ }
+
+ private GuestResources guestResources(NodeAccess access) {
+ String[] values = executeNode(access, "getconf _NPROCESSORS_ONLN; awk '/^MemTotal:/ {print $2}' /proc/meminfo").trim().split("\\s+");
+ if (values.length != 2) throw new CloudRuntimeException("Unable to read guest CPU and memory");
+ return new GuestResources(Long.parseLong(values[0]), Long.parseLong(values[1]));
+ }
+
+ private boolean guestMatchesTarget(GuestResources observed, ServiceOffering target) {
+ long targetMemory = target.getRamSize() * 1024L * 1024L; long observedMemory = observed.memoryKiB * 1024L;
+ long overhead = Math.max(targetMemory / 50L, MINIMUM_MEMORY_OVERHEAD_BYTES);
+ return observed.cpu == target.getCpu() && observedMemory <= targetMemory && targetMemory - observedMemory <= overhead;
+ }
+
+ private boolean kubernetesMatchesGuest(NodeCapacitySnapshot observed, NodeCapacitySnapshot before) {
+ return observed.isReady() && observed.getCapacityCpuMillis() == observed.getGuestOnlineCpuCount() * 1000L
+ && Math.abs(observed.getCapacityMemoryBytes() - observed.getGuestMemoryKiB() * 1024L) <= KUBERNETES_MEMORY_REPORTING_TOLERANCE_BYTES
+ && (observed.getGuestOnlineCpuCount() <= before.getGuestOnlineCpuCount() || observed.getAllocatableCpuMillis() > before.getAllocatableCpuMillis())
+ && (observed.getGuestMemoryKiB() <= before.getGuestMemoryKiB() || observed.getAllocatableMemoryBytes() > before.getAllocatableMemoryBytes());
+ }
+
+ protected String executeControl(NodeAccess access, String command) {
+ return execute(access.getControlAddress(), access.getControlPort(), access, command);
+ }
+
+ protected String executeNode(NodeAccess access, String command) {
+ return execute(access.getNodeAddress(), access.getNodePort(), access, command);
+ }
+
+ private String execute(String address, int port, NodeAccess access, String command) {
+ Pair<Boolean, String> result;
+ try {
+ result = SshHelper.sshExecute(address, port, access.getUser(), access.getSshKeyFile(), null, command, 10000, 10000, COMMAND_TIMEOUT_MS);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new NodeCapacityReconciliationException("CKS live-resize command interrupted", e);
+ } catch (RuntimeException e) {
+ throw e;
+ } catch (Exception e) {
+ // SshHelper declares Exception, including authentication failures.
+ // Translate only at this legacy boundary, not around the resize workflow.
+ throw new NodeCapacityReconciliationException("CKS live-resize SSH command failed", e);
+ }
+ if (!Boolean.TRUE.equals(result.first())) {
+ throw new NodeCapacityReconciliationException("CKS live-resize command failed");
+ }
+ return StringUtils.defaultString(result.second());
+ }
+ private String nodeName(UserVm vm) { String name = StringUtils.lowerCase(vm.getHostName()); if (!NODE_NAME.matcher(name).matches()) throw new CloudRuntimeException("Invalid Kubernetes node name for live resize"); return name; }
Review Comment:
nodeName calls StringUtils.lowerCase(vm.getHostName()) and passes the result to the regex. A null host name causes an NPE. Worth adding a null check to make it defensive.
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/utils/KubernetesClusterNodeCapacityReconcilerImpl.java:
##########
@@ -0,0 +1,232 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.kubernetes.cluster.utils;
+
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import com.cloud.kubernetes.cluster.KubernetesCluster;
+import com.cloud.kubernetes.cluster.KubernetesServiceHelper.KubernetesClusterNodeType;
+import com.cloud.offering.ServiceOffering;
+import com.cloud.uservm.UserVm;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.ssh.SshHelper;
+import com.google.gson.JsonArray;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParser;
+import org.apache.commons.lang3.StringUtils;
+
+/** SSH-backed implementation for a rolling CKS live-resize reconciliation. */
+public class KubernetesClusterNodeCapacityReconcilerImpl implements KubernetesClusterNodeCapacityReconciler {
+ // KVM QA observes 4,568,788 KiB in a 4,608 MiB guest after memory hotplug
+ // (about 146 MiB below the offering), so keep a small margin above that
+ // platform overhead while still rejecting materially undersized guests.
+ static final long MINIMUM_MEMORY_OVERHEAD_BYTES = 160L * 1024L * 1024L;
+ static final long KUBERNETES_MEMORY_REPORTING_TOLERANCE_BYTES = 16L * 1024L * 1024L;
+ private static final int COMMAND_TIMEOUT_MS = 30000;
+ private static final int POLL_INTERVAL_MS = 5000;
+ private static final Pattern NODE_NAME = Pattern.compile("[A-Za-z0-9][A-Za-z0-9.-]{0,252}");
+ private static final Pattern MEMORY_QUANTITY = Pattern.compile("([0-9]+)(Ki|Mi|Gi|Ti|K|M|G|T)?");
+ private static final String RESIZE_ANNOTATION = "cloudstack.apache.org/cks-live-resize";
+ static final String ONLINE_HOTPLUGGED_RESOURCES = "sudo sh -c 'for cpu in /sys/devices/system/cpu/cpu[0-9]*; do "
+ + "[ -e \"$cpu/online\" ] || continue; echo 1 > \"$cpu/online\"; done; "
+ + "for memory in /sys/devices/system/memory/memory[0-9]*; do "
+ + "[ -e \"$memory/online\" ] || continue; echo online > \"$memory/online\"; done'";
+
+ @Override
+ public boolean requiresKubeletRefresh(ServiceOffering oldOffering, ServiceOffering newOffering,
+ KubernetesClusterNodeType nodeType, KubernetesCluster.State originalState) {
+ return KubernetesCluster.State.Running == originalState
+ && (KubernetesClusterNodeType.WORKER == nodeType || KubernetesClusterNodeType.CONTROL == nodeType)
+ && capacityChanged(oldOffering, newOffering);
+ }
+
+ @Override
+ public NodeCapacitySnapshot captureBefore(KubernetesCluster cluster, UserVm vm, NodeAccess access) {
+ return nodeSnapshot(cluster, vm, access, guestResources(access));
+ }
+
+ @Override
+ public void cordonIfNeeded(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "=" + cluster.getUuid() + " --overwrite");
+ executeControl(access, "sudo /opt/bin/kubectl cordon " + node);
+ while (System.currentTimeMillis() < deadline) {
+ if (nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("CORDON", vm, "Kubernetes node did not become unschedulable");
+ }
+
+ @Override
+ public void onlineGuestResources(NodeAccess access) {
+ executeNode(access, ONLINE_HOTPLUGGED_RESOURCES);
+ }
+
+ @Override
+ public void verifyGuestResources(UserVm vm, ServiceOffering target, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ if (guestMatchesTarget(guestResources(access), target)) return;
+ sleep();
+ }
+ throw failure("GUEST_VERIFY", vm, "guest CPU or memory did not reach the target offering");
+ }
+
+ @Override
+ public boolean isKubernetesResourcesCurrent(NodeCapacitySnapshot snapshot, ServiceOffering target) {
+ GuestResources guest = new GuestResources(snapshot.getGuestOnlineCpuCount(), snapshot.getGuestMemoryKiB());
+ return guestMatchesTarget(guest, target) && kubernetesMatchesGuest(snapshot, snapshot);
+ }
+
+ @Override
+ public void restartKubelet(UserVm vm, NodeAccess access, long deadline) {
+ executeNode(access, "sudo systemctl restart kubelet");
+ while (System.currentTimeMillis() < deadline) {
+ if ("active".equals(executeNode(access, "sudo systemctl is-active kubelet").trim())) return;
+ sleep();
+ }
+ throw failure("KUBELET_RESTART", vm, "kubelet did not become active");
+ }
+
+ @Override
+ public NodeCapacitySnapshot waitForKubernetesResources(KubernetesCluster cluster, UserVm vm, ServiceOffering target,
+ NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ GuestResources guest = guestResources(access);
+ NodeCapacitySnapshot observed = nodeSnapshot(cluster, vm, access, guest);
+ if (guestMatchesTarget(guest, target) && kubernetesMatchesGuest(observed, before)) return observed;
+ sleep();
+ }
+ throw failure("CAPACITY_VERIFY", vm, "Kubernetes capacity did not match the resized guest");
+ }
+
+ @Override
+ public void restoreSchedulability(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl uncordon " + node);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "-");
+ while (System.currentTimeMillis() < deadline) {
+ if (!nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("RESTORE_SCHEDULABILITY", vm, "Kubernetes node remained unschedulable");
+ }
+
+ public static boolean capacityChanged(ServiceOffering oldOffering, ServiceOffering targetOffering) {
+ return oldOffering != null && targetOffering != null
+ && (!Objects.equals(oldOffering.getCpu(), targetOffering.getCpu())
+ || !Objects.equals(oldOffering.getRamSize(), targetOffering.getRamSize()));
+ }
+
+ static long parseCpuMillis(String value) { return value.endsWith("m") ? Long.parseLong(value.substring(0, value.length() - 1)) : Long.parseLong(value) * 1000L; }
+ static long parseMemoryBytes(String value) {
+ Matcher matcher = MEMORY_QUANTITY.matcher(value);
+ if (!matcher.matches()) throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ long number = Long.parseLong(matcher.group(1)); String unit = matcher.group(2);
+ if (unit == null) return number;
+ switch (unit) {
+ case "Ki": return number * 1024L; case "Mi": return number * 1024L * 1024L; case "Gi": return number * 1024L * 1024L * 1024L;
+ case "Ti": return number * 1024L * 1024L * 1024L * 1024L; case "K": return number * 1000L; case "M": return number * 1000L * 1000L;
+ case "G": return number * 1000L * 1000L * 1000L; case "T": return number * 1000L * 1000L * 1000L * 1000L;
+ default: throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ }
+ }
+
+ private NodeCapacitySnapshot nodeSnapshot(KubernetesCluster cluster, UserVm vm, NodeAccess access, GuestResources guest) {
+ JsonObject node = new JsonParser().parse(executeControl(access, "sudo /opt/bin/kubectl get node " + nodeName(vm) + " -o json")).getAsJsonObject();
Review Comment:
`JsonParser().parse(..)` is deprecated, can we use the right one
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterScaleWorker.java:
##########
@@ -376,24 +377,171 @@ private void scaleKubernetesClusterOffering(KubernetesClusterNodeType nodeType,
List<KubernetesClusterVmMapVO> vmList = kubernetesClusterVmMapDao.listByClusterIdAndVmType(kubernetesCluster.getId(), nodeType);
final long tobeScaledVMCount = Math.min(vmList.size(), size);
for (long i = 0; i < tobeScaledVMCount; i++) {
- KubernetesClusterVmMapVO vmMapVO = vmList.get((int) i);
- UserVmVO userVM = userVmDao.findById(vmMapVO.getVmId());
- boolean result = false;
- try {
- result = userVmManager.upgradeVirtualMachine(userVM.getId(), serviceOffering.getId(), new HashMap<String, String>());
- } catch (RuntimeException | ResourceUnavailableException | ManagementServerException | VirtualMachineMigrationException e) {
- logTransitStateAndThrow(Level.ERROR, String.format("Scaling Kubernetes cluster : %s failed, unable to scale cluster VM : %s due to %s", kubernetesCluster.getName(), userVM.getDisplayName(), e.getMessage()), kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed, e);
- }
- if (!result) {
- logTransitStateAndThrow(Level.WARN, String.format("Scaling Kubernetes cluster : %s failed, unable to scale cluster VM : %s", kubernetesCluster.getName(), userVM.getDisplayName()),kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
- }
+ new NodeOfferingResize(vmList.get((int) i), nodeType, serviceOffering).execute();
if (System.currentTimeMillis() > scaleTimeoutTime) {
logTransitStateAndThrow(Level.WARN, String.format("Scaling Kubernetes cluster : %s failed, scaling action timed out", kubernetesCluster.getName()),kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
}
}
kubernetesCluster = updateKubernetesClusterEntryForNodeType(null, nodeType, serviceOffering, updateNodeOffering, updateClusterOffering);
}
+ private final class NodeOfferingResize {
+ private final KubernetesClusterVmMapVO vmMap;
+ private final KubernetesClusterNodeType nodeType;
+ private final ServiceOffering targetOffering;
+ private final ServiceOffering oldOffering;
+ private UserVmVO vm;
+ private NodeAccess access;
+ private NodeCapacitySnapshot before;
+ private boolean reconcileKubernetes;
+ private boolean cksCordonCompleted;
+ private boolean resizeSucceeded;
+
+ private NodeOfferingResize(KubernetesClusterVmMapVO vmMap, KubernetesClusterNodeType nodeType, ServiceOffering targetOffering) {
+ this.vmMap = vmMap;
+ this.nodeType = nodeType;
+ this.targetOffering = targetOffering;
+ vm = userVmDao.findById(vmMap.getVmId());
+ if (vm == null) {
+ logTransitStateAndThrow(Level.ERROR, String.format("Scaling Kubernetes cluster : %s failed, unable to find cluster VM %s",
+ kubernetesCluster.getName(), vmMap.getVmId()), kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
+ }
+ oldOffering = serviceOfferingDao.findById(vm.getServiceOfferingId());
+ resizeSucceeded = Objects.equals(targetOffering.getId(), vm.getServiceOfferingId());
+ }
+
+ private void execute() {
+ validateLiveResize();
+ try {
+ prepareReconciliation();
+ resizeVm();
+ reconcileCapacity();
+ } catch (ResourceUnavailableException | ManagementServerException | VirtualMachineMigrationException | CloudRuntimeException e) {
+ restoreAfterFailedResize();
+ reportFailure(e);
+ }
+ }
+
+ private void validateLiveResize() {
+ if (KubernetesClusterNodeCapacityReconcilerImpl.capacityChanged(oldOffering, targetOffering)
+ && KubernetesCluster.State.Running == originalState && ETCD == nodeType
+ && KubernetesCluster.ClusterType.CloudManaged == kubernetesCluster.getClusterType()) {
+ logTransitStateAndThrow(Level.ERROR, String.format("Scaling Kubernetes cluster : %s cannot live-resize dedicated etcd VM %s; " +
+ "etcd health reconciliation is not implemented", kubernetesCluster.getName(), vm.getDisplayName()),
+ kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
+ }
+ }
+
+ private void prepareReconciliation() {
+ reconcileKubernetes = shouldReconcileNodeCapacity(vmMap, vm, oldOffering, targetOffering, nodeType);
+ if (reconcileKubernetes || isReconciliationRetryCandidate(vmMap, vm, targetOffering, nodeType)) {
+ access = resolveNodeAccess(vm);
+ before = kubernetesClusterNodeCapacityReconciler.captureBefore(kubernetesCluster, vm, access);
Review Comment:
For a node already at the target offering (isReconciliationRetryCandidate), only check for a leftover CKS cordon. If resolveNodeAccess or captureBefore fails here (no port-forwarding rule, SSH unreachable), the whole scale fails even though this node needs no work. Can we catch CloudRuntimeException here and log a warning and skip the node?
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterScaleWorker.java:
##########
@@ -376,24 +377,171 @@ private void scaleKubernetesClusterOffering(KubernetesClusterNodeType nodeType,
List<KubernetesClusterVmMapVO> vmList = kubernetesClusterVmMapDao.listByClusterIdAndVmType(kubernetesCluster.getId(), nodeType);
final long tobeScaledVMCount = Math.min(vmList.size(), size);
for (long i = 0; i < tobeScaledVMCount; i++) {
- KubernetesClusterVmMapVO vmMapVO = vmList.get((int) i);
- UserVmVO userVM = userVmDao.findById(vmMapVO.getVmId());
- boolean result = false;
- try {
- result = userVmManager.upgradeVirtualMachine(userVM.getId(), serviceOffering.getId(), new HashMap<String, String>());
- } catch (RuntimeException | ResourceUnavailableException | ManagementServerException | VirtualMachineMigrationException e) {
- logTransitStateAndThrow(Level.ERROR, String.format("Scaling Kubernetes cluster : %s failed, unable to scale cluster VM : %s due to %s", kubernetesCluster.getName(), userVM.getDisplayName(), e.getMessage()), kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed, e);
- }
- if (!result) {
- logTransitStateAndThrow(Level.WARN, String.format("Scaling Kubernetes cluster : %s failed, unable to scale cluster VM : %s", kubernetesCluster.getName(), userVM.getDisplayName()),kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
- }
+ new NodeOfferingResize(vmList.get((int) i), nodeType, serviceOffering).execute();
Review Comment:
Each node now waits on several steps (cordon, guest check, kubelet restart, capacity check, uncordon), and all of them share the one scale timeout. On a bigger cluster, the later nodes may be left with very little time and fail with a timeout. Is KubernetesClusterScaleTimeout enough for this, or should we give each node its own timeout?
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/utils/KubernetesClusterNodeCapacityReconcilerImpl.java:
##########
@@ -0,0 +1,232 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.kubernetes.cluster.utils;
+
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import com.cloud.kubernetes.cluster.KubernetesCluster;
+import com.cloud.kubernetes.cluster.KubernetesServiceHelper.KubernetesClusterNodeType;
+import com.cloud.offering.ServiceOffering;
+import com.cloud.uservm.UserVm;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.ssh.SshHelper;
+import com.google.gson.JsonArray;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParser;
+import org.apache.commons.lang3.StringUtils;
+
+/** SSH-backed implementation for a rolling CKS live-resize reconciliation. */
+public class KubernetesClusterNodeCapacityReconcilerImpl implements KubernetesClusterNodeCapacityReconciler {
+ // KVM QA observes 4,568,788 KiB in a 4,608 MiB guest after memory hotplug
+ // (about 146 MiB below the offering), so keep a small margin above that
+ // platform overhead while still rejecting materially undersized guests.
+ static final long MINIMUM_MEMORY_OVERHEAD_BYTES = 160L * 1024L * 1024L;
+ static final long KUBERNETES_MEMORY_REPORTING_TOLERANCE_BYTES = 16L * 1024L * 1024L;
+ private static final int COMMAND_TIMEOUT_MS = 30000;
+ private static final int POLL_INTERVAL_MS = 5000;
+ private static final Pattern NODE_NAME = Pattern.compile("[A-Za-z0-9][A-Za-z0-9.-]{0,252}");
+ private static final Pattern MEMORY_QUANTITY = Pattern.compile("([0-9]+)(Ki|Mi|Gi|Ti|K|M|G|T)?");
+ private static final String RESIZE_ANNOTATION = "cloudstack.apache.org/cks-live-resize";
+ static final String ONLINE_HOTPLUGGED_RESOURCES = "sudo sh -c 'for cpu in /sys/devices/system/cpu/cpu[0-9]*; do "
+ + "[ -e \"$cpu/online\" ] || continue; echo 1 > \"$cpu/online\"; done; "
+ + "for memory in /sys/devices/system/memory/memory[0-9]*; do "
+ + "[ -e \"$memory/online\" ] || continue; echo online > \"$memory/online\"; done'";
+
+ @Override
+ public boolean requiresKubeletRefresh(ServiceOffering oldOffering, ServiceOffering newOffering,
+ KubernetesClusterNodeType nodeType, KubernetesCluster.State originalState) {
+ return KubernetesCluster.State.Running == originalState
+ && (KubernetesClusterNodeType.WORKER == nodeType || KubernetesClusterNodeType.CONTROL == nodeType)
+ && capacityChanged(oldOffering, newOffering);
+ }
+
+ @Override
+ public NodeCapacitySnapshot captureBefore(KubernetesCluster cluster, UserVm vm, NodeAccess access) {
+ return nodeSnapshot(cluster, vm, access, guestResources(access));
+ }
+
+ @Override
+ public void cordonIfNeeded(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "=" + cluster.getUuid() + " --overwrite");
+ executeControl(access, "sudo /opt/bin/kubectl cordon " + node);
+ while (System.currentTimeMillis() < deadline) {
+ if (nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("CORDON", vm, "Kubernetes node did not become unschedulable");
+ }
+
+ @Override
+ public void onlineGuestResources(NodeAccess access) {
+ executeNode(access, ONLINE_HOTPLUGGED_RESOURCES);
+ }
+
+ @Override
+ public void verifyGuestResources(UserVm vm, ServiceOffering target, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ if (guestMatchesTarget(guestResources(access), target)) return;
+ sleep();
+ }
+ throw failure("GUEST_VERIFY", vm, "guest CPU or memory did not reach the target offering");
+ }
+
+ @Override
+ public boolean isKubernetesResourcesCurrent(NodeCapacitySnapshot snapshot, ServiceOffering target) {
+ GuestResources guest = new GuestResources(snapshot.getGuestOnlineCpuCount(), snapshot.getGuestMemoryKiB());
+ return guestMatchesTarget(guest, target) && kubernetesMatchesGuest(snapshot, snapshot);
+ }
+
+ @Override
+ public void restartKubelet(UserVm vm, NodeAccess access, long deadline) {
+ executeNode(access, "sudo systemctl restart kubelet");
+ while (System.currentTimeMillis() < deadline) {
+ if ("active".equals(executeNode(access, "sudo systemctl is-active kubelet").trim())) return;
+ sleep();
+ }
+ throw failure("KUBELET_RESTART", vm, "kubelet did not become active");
+ }
+
+ @Override
+ public NodeCapacitySnapshot waitForKubernetesResources(KubernetesCluster cluster, UserVm vm, ServiceOffering target,
+ NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ GuestResources guest = guestResources(access);
+ NodeCapacitySnapshot observed = nodeSnapshot(cluster, vm, access, guest);
+ if (guestMatchesTarget(guest, target) && kubernetesMatchesGuest(observed, before)) return observed;
+ sleep();
+ }
+ throw failure("CAPACITY_VERIFY", vm, "Kubernetes capacity did not match the resized guest");
+ }
+
+ @Override
+ public void restoreSchedulability(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl uncordon " + node);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "-");
+ while (System.currentTimeMillis() < deadline) {
+ if (!nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("RESTORE_SCHEDULABILITY", vm, "Kubernetes node remained unschedulable");
+ }
+
+ public static boolean capacityChanged(ServiceOffering oldOffering, ServiceOffering targetOffering) {
+ return oldOffering != null && targetOffering != null
+ && (!Objects.equals(oldOffering.getCpu(), targetOffering.getCpu())
+ || !Objects.equals(oldOffering.getRamSize(), targetOffering.getRamSize()));
+ }
+
+ static long parseCpuMillis(String value) { return value.endsWith("m") ? Long.parseLong(value.substring(0, value.length() - 1)) : Long.parseLong(value) * 1000L; }
+ static long parseMemoryBytes(String value) {
+ Matcher matcher = MEMORY_QUANTITY.matcher(value);
+ if (!matcher.matches()) throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ long number = Long.parseLong(matcher.group(1)); String unit = matcher.group(2);
+ if (unit == null) return number;
+ switch (unit) {
+ case "Ki": return number * 1024L; case "Mi": return number * 1024L * 1024L; case "Gi": return number * 1024L * 1024L * 1024L;
+ case "Ti": return number * 1024L * 1024L * 1024L * 1024L; case "K": return number * 1000L; case "M": return number * 1000L * 1000L;
+ case "G": return number * 1000L * 1000L * 1000L; case "T": return number * 1000L * 1000L * 1000L * 1000L;
+ default: throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ }
+ }
+
+ private NodeCapacitySnapshot nodeSnapshot(KubernetesCluster cluster, UserVm vm, NodeAccess access, GuestResources guest) {
+ JsonObject node = new JsonParser().parse(executeControl(access, "sudo /opt/bin/kubectl get node " + nodeName(vm) + " -o json")).getAsJsonObject();
+ JsonObject annotations = node.getAsJsonObject("metadata").has("annotations") ? node.getAsJsonObject("metadata").getAsJsonObject("annotations") : null;
+ boolean cksCordon = annotations != null && annotations.has(RESIZE_ANNOTATION) && cluster.getUuid().equals(annotations.get(RESIZE_ANNOTATION).getAsString());
+ JsonObject spec = node.has("spec") ? node.getAsJsonObject("spec") : new JsonObject(); JsonObject status = node.getAsJsonObject("status");
+ JsonObject capacity = status.getAsJsonObject("capacity"); JsonObject allocatable = status.getAsJsonObject("allocatable");
Review Comment:
same here for allocatable
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/utils/KubernetesClusterNodeCapacityReconcilerImpl.java:
##########
@@ -0,0 +1,232 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.kubernetes.cluster.utils;
+
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import com.cloud.kubernetes.cluster.KubernetesCluster;
+import com.cloud.kubernetes.cluster.KubernetesServiceHelper.KubernetesClusterNodeType;
+import com.cloud.offering.ServiceOffering;
+import com.cloud.uservm.UserVm;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.ssh.SshHelper;
+import com.google.gson.JsonArray;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParser;
+import org.apache.commons.lang3.StringUtils;
+
+/** SSH-backed implementation for a rolling CKS live-resize reconciliation. */
+public class KubernetesClusterNodeCapacityReconcilerImpl implements KubernetesClusterNodeCapacityReconciler {
+ // KVM QA observes 4,568,788 KiB in a 4,608 MiB guest after memory hotplug
+ // (about 146 MiB below the offering), so keep a small margin above that
+ // platform overhead while still rejecting materially undersized guests.
+ static final long MINIMUM_MEMORY_OVERHEAD_BYTES = 160L * 1024L * 1024L;
+ static final long KUBERNETES_MEMORY_REPORTING_TOLERANCE_BYTES = 16L * 1024L * 1024L;
+ private static final int COMMAND_TIMEOUT_MS = 30000;
+ private static final int POLL_INTERVAL_MS = 5000;
+ private static final Pattern NODE_NAME = Pattern.compile("[A-Za-z0-9][A-Za-z0-9.-]{0,252}");
+ private static final Pattern MEMORY_QUANTITY = Pattern.compile("([0-9]+)(Ki|Mi|Gi|Ti|K|M|G|T)?");
+ private static final String RESIZE_ANNOTATION = "cloudstack.apache.org/cks-live-resize";
+ static final String ONLINE_HOTPLUGGED_RESOURCES = "sudo sh -c 'for cpu in /sys/devices/system/cpu/cpu[0-9]*; do "
+ + "[ -e \"$cpu/online\" ] || continue; echo 1 > \"$cpu/online\"; done; "
+ + "for memory in /sys/devices/system/memory/memory[0-9]*; do "
+ + "[ -e \"$memory/online\" ] || continue; echo online > \"$memory/online\"; done'";
+
+ @Override
+ public boolean requiresKubeletRefresh(ServiceOffering oldOffering, ServiceOffering newOffering,
+ KubernetesClusterNodeType nodeType, KubernetesCluster.State originalState) {
+ return KubernetesCluster.State.Running == originalState
+ && (KubernetesClusterNodeType.WORKER == nodeType || KubernetesClusterNodeType.CONTROL == nodeType)
+ && capacityChanged(oldOffering, newOffering);
+ }
+
+ @Override
+ public NodeCapacitySnapshot captureBefore(KubernetesCluster cluster, UserVm vm, NodeAccess access) {
+ return nodeSnapshot(cluster, vm, access, guestResources(access));
+ }
+
+ @Override
+ public void cordonIfNeeded(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "=" + cluster.getUuid() + " --overwrite");
+ executeControl(access, "sudo /opt/bin/kubectl cordon " + node);
+ while (System.currentTimeMillis() < deadline) {
+ if (nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("CORDON", vm, "Kubernetes node did not become unschedulable");
+ }
+
+ @Override
+ public void onlineGuestResources(NodeAccess access) {
+ executeNode(access, ONLINE_HOTPLUGGED_RESOURCES);
+ }
+
+ @Override
+ public void verifyGuestResources(UserVm vm, ServiceOffering target, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ if (guestMatchesTarget(guestResources(access), target)) return;
+ sleep();
+ }
+ throw failure("GUEST_VERIFY", vm, "guest CPU or memory did not reach the target offering");
+ }
+
+ @Override
+ public boolean isKubernetesResourcesCurrent(NodeCapacitySnapshot snapshot, ServiceOffering target) {
+ GuestResources guest = new GuestResources(snapshot.getGuestOnlineCpuCount(), snapshot.getGuestMemoryKiB());
+ return guestMatchesTarget(guest, target) && kubernetesMatchesGuest(snapshot, snapshot);
+ }
+
+ @Override
+ public void restartKubelet(UserVm vm, NodeAccess access, long deadline) {
+ executeNode(access, "sudo systemctl restart kubelet");
+ while (System.currentTimeMillis() < deadline) {
+ if ("active".equals(executeNode(access, "sudo systemctl is-active kubelet").trim())) return;
+ sleep();
+ }
+ throw failure("KUBELET_RESTART", vm, "kubelet did not become active");
+ }
+
+ @Override
+ public NodeCapacitySnapshot waitForKubernetesResources(KubernetesCluster cluster, UserVm vm, ServiceOffering target,
+ NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ GuestResources guest = guestResources(access);
+ NodeCapacitySnapshot observed = nodeSnapshot(cluster, vm, access, guest);
+ if (guestMatchesTarget(guest, target) && kubernetesMatchesGuest(observed, before)) return observed;
+ sleep();
+ }
+ throw failure("CAPACITY_VERIFY", vm, "Kubernetes capacity did not match the resized guest");
+ }
+
+ @Override
+ public void restoreSchedulability(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl uncordon " + node);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "-");
+ while (System.currentTimeMillis() < deadline) {
+ if (!nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("RESTORE_SCHEDULABILITY", vm, "Kubernetes node remained unschedulable");
+ }
+
+ public static boolean capacityChanged(ServiceOffering oldOffering, ServiceOffering targetOffering) {
+ return oldOffering != null && targetOffering != null
+ && (!Objects.equals(oldOffering.getCpu(), targetOffering.getCpu())
+ || !Objects.equals(oldOffering.getRamSize(), targetOffering.getRamSize()));
+ }
+
+ static long parseCpuMillis(String value) { return value.endsWith("m") ? Long.parseLong(value.substring(0, value.length() - 1)) : Long.parseLong(value) * 1000L; }
+ static long parseMemoryBytes(String value) {
+ Matcher matcher = MEMORY_QUANTITY.matcher(value);
+ if (!matcher.matches()) throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ long number = Long.parseLong(matcher.group(1)); String unit = matcher.group(2);
+ if (unit == null) return number;
+ switch (unit) {
+ case "Ki": return number * 1024L; case "Mi": return number * 1024L * 1024L; case "Gi": return number * 1024L * 1024L * 1024L;
+ case "Ti": return number * 1024L * 1024L * 1024L * 1024L; case "K": return number * 1000L; case "M": return number * 1000L * 1000L;
+ case "G": return number * 1000L * 1000L * 1000L; case "T": return number * 1000L * 1000L * 1000L * 1000L;
+ default: throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ }
+ }
+
+ private NodeCapacitySnapshot nodeSnapshot(KubernetesCluster cluster, UserVm vm, NodeAccess access, GuestResources guest) {
+ JsonObject node = new JsonParser().parse(executeControl(access, "sudo /opt/bin/kubectl get node " + nodeName(vm) + " -o json")).getAsJsonObject();
+ JsonObject annotations = node.getAsJsonObject("metadata").has("annotations") ? node.getAsJsonObject("metadata").getAsJsonObject("annotations") : null;
+ boolean cksCordon = annotations != null && annotations.has(RESIZE_ANNOTATION) && cluster.getUuid().equals(annotations.get(RESIZE_ANNOTATION).getAsString());
+ JsonObject spec = node.has("spec") ? node.getAsJsonObject("spec") : new JsonObject(); JsonObject status = node.getAsJsonObject("status");
+ JsonObject capacity = status.getAsJsonObject("capacity"); JsonObject allocatable = status.getAsJsonObject("allocatable");
+ return new NodeCapacitySnapshot(spec.has("unschedulable") && spec.get("unschedulable").getAsBoolean(), cksCordon,
+ guest == null ? 0 : guest.cpu, guest == null ? 0 : guest.memoryKiB, parseCpuMillis(capacity.get("cpu").getAsString()),
+ parseMemoryBytes(capacity.get("memory").getAsString()), parseCpuMillis(allocatable.get("cpu").getAsString()),
+ parseMemoryBytes(allocatable.get("memory").getAsString()), isReady(status));
Review Comment:
```suggestion
return new NodeCapacitySnapshot(spec.has("unschedulable") && spec.get("unschedulable").getAsBoolean(),
cksCordon,
guest == null ? 0 : guest.cpu,
guest == null ? 0 : guest.memoryKiB,
parseCpuMillis(capacity.get("cpu").getAsString()),
parseMemoryBytes(capacity.get("memory").getAsString()),
parseCpuMillis(allocatable.get("cpu").getAsString()),
parseMemoryBytes(allocatable.get("memory").getAsString()),
isReady(status));
```
Something like this for better readability - as it has some expressions and its hard to read. Thanks
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterScaleWorker.java:
##########
@@ -376,24 +377,171 @@ private void scaleKubernetesClusterOffering(KubernetesClusterNodeType nodeType,
List<KubernetesClusterVmMapVO> vmList = kubernetesClusterVmMapDao.listByClusterIdAndVmType(kubernetesCluster.getId(), nodeType);
final long tobeScaledVMCount = Math.min(vmList.size(), size);
for (long i = 0; i < tobeScaledVMCount; i++) {
- KubernetesClusterVmMapVO vmMapVO = vmList.get((int) i);
- UserVmVO userVM = userVmDao.findById(vmMapVO.getVmId());
- boolean result = false;
- try {
- result = userVmManager.upgradeVirtualMachine(userVM.getId(), serviceOffering.getId(), new HashMap<String, String>());
- } catch (RuntimeException | ResourceUnavailableException | ManagementServerException | VirtualMachineMigrationException e) {
- logTransitStateAndThrow(Level.ERROR, String.format("Scaling Kubernetes cluster : %s failed, unable to scale cluster VM : %s due to %s", kubernetesCluster.getName(), userVM.getDisplayName(), e.getMessage()), kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed, e);
- }
- if (!result) {
- logTransitStateAndThrow(Level.WARN, String.format("Scaling Kubernetes cluster : %s failed, unable to scale cluster VM : %s", kubernetesCluster.getName(), userVM.getDisplayName()),kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
- }
+ new NodeOfferingResize(vmList.get((int) i), nodeType, serviceOffering).execute();
if (System.currentTimeMillis() > scaleTimeoutTime) {
logTransitStateAndThrow(Level.WARN, String.format("Scaling Kubernetes cluster : %s failed, scaling action timed out", kubernetesCluster.getName()),kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
}
}
kubernetesCluster = updateKubernetesClusterEntryForNodeType(null, nodeType, serviceOffering, updateNodeOffering, updateClusterOffering);
}
+ private final class NodeOfferingResize {
+ private final KubernetesClusterVmMapVO vmMap;
+ private final KubernetesClusterNodeType nodeType;
+ private final ServiceOffering targetOffering;
+ private final ServiceOffering oldOffering;
+ private UserVmVO vm;
+ private NodeAccess access;
+ private NodeCapacitySnapshot before;
+ private boolean reconcileKubernetes;
+ private boolean cksCordonCompleted;
+ private boolean resizeSucceeded;
+
+ private NodeOfferingResize(KubernetesClusterVmMapVO vmMap, KubernetesClusterNodeType nodeType, ServiceOffering targetOffering) {
+ this.vmMap = vmMap;
+ this.nodeType = nodeType;
+ this.targetOffering = targetOffering;
+ vm = userVmDao.findById(vmMap.getVmId());
+ if (vm == null) {
+ logTransitStateAndThrow(Level.ERROR, String.format("Scaling Kubernetes cluster : %s failed, unable to find cluster VM %s",
+ kubernetesCluster.getName(), vmMap.getVmId()), kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
+ }
+ oldOffering = serviceOfferingDao.findById(vm.getServiceOfferingId());
+ resizeSucceeded = Objects.equals(targetOffering.getId(), vm.getServiceOfferingId());
+ }
+
+ private void execute() {
+ validateLiveResize();
+ try {
+ prepareReconciliation();
+ resizeVm();
+ reconcileCapacity();
+ } catch (ResourceUnavailableException | ManagementServerException | VirtualMachineMigrationException | CloudRuntimeException e) {
+ restoreAfterFailedResize();
+ reportFailure(e);
+ }
+ }
+
+ private void validateLiveResize() {
+ if (KubernetesClusterNodeCapacityReconcilerImpl.capacityChanged(oldOffering, targetOffering)
+ && KubernetesCluster.State.Running == originalState && ETCD == nodeType
+ && KubernetesCluster.ClusterType.CloudManaged == kubernetesCluster.getClusterType()) {
+ logTransitStateAndThrow(Level.ERROR, String.format("Scaling Kubernetes cluster : %s cannot live-resize dedicated etcd VM %s; " +
+ "etcd health reconciliation is not implemented", kubernetesCluster.getName(), vm.getDisplayName()),
+ kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
+ }
+ }
+
+ private void prepareReconciliation() {
+ reconcileKubernetes = shouldReconcileNodeCapacity(vmMap, vm, oldOffering, targetOffering, nodeType);
Review Comment:
When reconciliation is skipped (external node, not CloudManaged, not KVM, cluster not Running, or a node type other than control/worker), nothing is logged. The node gets resized but Kubernetes isn't reconciled, and the user has no way to tell. Can we add a log line saying why it was skipped?
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/utils/KubernetesClusterNodeCapacityReconciler.java:
##########
@@ -0,0 +1,68 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.kubernetes.cluster.utils;
+
+import java.io.File;
+
+import com.cloud.kubernetes.cluster.KubernetesCluster;
+import com.cloud.kubernetes.cluster.KubernetesServiceHelper.KubernetesClusterNodeType;
+import com.cloud.offering.ServiceOffering;
+import com.cloud.uservm.UserVm;
+
+/** Reconciles the guest and Kubernetes resource views after a live VM resize. */
+public interface KubernetesClusterNodeCapacityReconciler {
Review Comment:
There is one implementation and the worker calls new. Nothing is injected, and the Spring context has no reconciler bean. Drop the interface and keep one class. And can probably move NodeAccess and NodeCapacitySnapshot to top-level classes.
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/utils/KubernetesClusterNodeCapacityReconciler.java:
##########
@@ -0,0 +1,68 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.kubernetes.cluster.utils;
+
+import java.io.File;
+
+import com.cloud.kubernetes.cluster.KubernetesCluster;
+import com.cloud.kubernetes.cluster.KubernetesServiceHelper.KubernetesClusterNodeType;
+import com.cloud.offering.ServiceOffering;
+import com.cloud.uservm.UserVm;
+
+/** Reconciles the guest and Kubernetes resource views after a live VM resize. */
+public interface KubernetesClusterNodeCapacityReconciler {
+ NodeCapacitySnapshot captureBefore(KubernetesCluster cluster, UserVm vm, NodeAccess access);
+ boolean requiresKubeletRefresh(ServiceOffering oldOffering, ServiceOffering newOffering,
+ KubernetesClusterNodeType nodeType, KubernetesCluster.State originalState);
+ void cordonIfNeeded(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline);
+ void onlineGuestResources(NodeAccess access);
+ void verifyGuestResources(UserVm vm, ServiceOffering target, NodeCapacitySnapshot before, NodeAccess access, long deadline);
+ boolean isKubernetesResourcesCurrent(NodeCapacitySnapshot snapshot, ServiceOffering target);
+ void restartKubelet(UserVm vm, NodeAccess access, long deadline);
+ NodeCapacitySnapshot waitForKubernetesResources(KubernetesCluster cluster, UserVm vm, ServiceOffering target,
+ NodeCapacitySnapshot before, NodeAccess access, long deadline);
+ void restoreSchedulability(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline);
+
+ final class NodeAccess {
Review Comment:
Many statements and fields are packed on one line. Reformat one per line. for both NodeAccess and NodeCapacitySnapshot classes
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterScaleWorker.java:
##########
@@ -376,24 +377,171 @@ private void scaleKubernetesClusterOffering(KubernetesClusterNodeType nodeType,
List<KubernetesClusterVmMapVO> vmList = kubernetesClusterVmMapDao.listByClusterIdAndVmType(kubernetesCluster.getId(), nodeType);
final long tobeScaledVMCount = Math.min(vmList.size(), size);
for (long i = 0; i < tobeScaledVMCount; i++) {
- KubernetesClusterVmMapVO vmMapVO = vmList.get((int) i);
- UserVmVO userVM = userVmDao.findById(vmMapVO.getVmId());
- boolean result = false;
- try {
- result = userVmManager.upgradeVirtualMachine(userVM.getId(), serviceOffering.getId(), new HashMap<String, String>());
- } catch (RuntimeException | ResourceUnavailableException | ManagementServerException | VirtualMachineMigrationException e) {
- logTransitStateAndThrow(Level.ERROR, String.format("Scaling Kubernetes cluster : %s failed, unable to scale cluster VM : %s due to %s", kubernetesCluster.getName(), userVM.getDisplayName(), e.getMessage()), kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed, e);
- }
- if (!result) {
- logTransitStateAndThrow(Level.WARN, String.format("Scaling Kubernetes cluster : %s failed, unable to scale cluster VM : %s", kubernetesCluster.getName(), userVM.getDisplayName()),kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
- }
+ new NodeOfferingResize(vmList.get((int) i), nodeType, serviceOffering).execute();
if (System.currentTimeMillis() > scaleTimeoutTime) {
logTransitStateAndThrow(Level.WARN, String.format("Scaling Kubernetes cluster : %s failed, scaling action timed out", kubernetesCluster.getName()),kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
}
}
kubernetesCluster = updateKubernetesClusterEntryForNodeType(null, nodeType, serviceOffering, updateNodeOffering, updateClusterOffering);
}
+ private final class NodeOfferingResize {
+ private final KubernetesClusterVmMapVO vmMap;
+ private final KubernetesClusterNodeType nodeType;
+ private final ServiceOffering targetOffering;
+ private final ServiceOffering oldOffering;
+ private UserVmVO vm;
+ private NodeAccess access;
+ private NodeCapacitySnapshot before;
+ private boolean reconcileKubernetes;
+ private boolean cksCordonCompleted;
+ private boolean resizeSucceeded;
+
+ private NodeOfferingResize(KubernetesClusterVmMapVO vmMap, KubernetesClusterNodeType nodeType, ServiceOffering targetOffering) {
+ this.vmMap = vmMap;
+ this.nodeType = nodeType;
+ this.targetOffering = targetOffering;
+ vm = userVmDao.findById(vmMap.getVmId());
+ if (vm == null) {
+ logTransitStateAndThrow(Level.ERROR, String.format("Scaling Kubernetes cluster : %s failed, unable to find cluster VM %s",
+ kubernetesCluster.getName(), vmMap.getVmId()), kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
+ }
+ oldOffering = serviceOfferingDao.findById(vm.getServiceOfferingId());
+ resizeSucceeded = Objects.equals(targetOffering.getId(), vm.getServiceOfferingId());
+ }
+
+ private void execute() {
+ validateLiveResize();
+ try {
+ prepareReconciliation();
+ resizeVm();
+ reconcileCapacity();
+ } catch (ResourceUnavailableException | ManagementServerException | VirtualMachineMigrationException | CloudRuntimeException e) {
+ restoreAfterFailedResize();
+ reportFailure(e);
+ }
+ }
+
+ private void validateLiveResize() {
+ if (KubernetesClusterNodeCapacityReconcilerImpl.capacityChanged(oldOffering, targetOffering)
+ && KubernetesCluster.State.Running == originalState && ETCD == nodeType
+ && KubernetesCluster.ClusterType.CloudManaged == kubernetesCluster.getClusterType()) {
+ logTransitStateAndThrow(Level.ERROR, String.format("Scaling Kubernetes cluster : %s cannot live-resize dedicated etcd VM %s; " +
+ "etcd health reconciliation is not implemented", kubernetesCluster.getName(), vm.getDisplayName()),
+ kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
+ }
+ }
+
+ private void prepareReconciliation() {
+ reconcileKubernetes = shouldReconcileNodeCapacity(vmMap, vm, oldOffering, targetOffering, nodeType);
+ if (reconcileKubernetes || isReconciliationRetryCandidate(vmMap, vm, targetOffering, nodeType)) {
+ access = resolveNodeAccess(vm);
+ before = kubernetesClusterNodeCapacityReconciler.captureBefore(kubernetesCluster, vm, access);
+ reconcileKubernetes = reconcileKubernetes || before.isCloudStackResizeCordon();
+ }
+ if (reconcileKubernetes) {
+ kubernetesClusterNodeCapacityReconciler.cordonIfNeeded(kubernetesCluster, vm, before, access, scaleTimeoutTime);
+ cksCordonCompleted = !before.isUnschedulable() || before.isCloudStackResizeCordon();
+ }
+ }
+
+ private void resizeVm() throws ResourceUnavailableException, ManagementServerException, VirtualMachineMigrationException {
+ if (resizeSucceeded) {
+ return;
+ }
+ if (!userVmManager.upgradeVirtualMachine(vm.getId(), targetOffering.getId(), new HashMap<>())) {
+ throw new CloudRuntimeException("Unable to scale cluster VM " + vm.getDisplayName());
+ }
+ resizeSucceeded = true;
+ UserVmVO resizedVm = userVmDao.findById(vm.getId());
+ if (resizedVm == null || !Objects.equals(targetOffering.getId(), resizedVm.getServiceOfferingId())) {
+ throw new CloudRuntimeException("VM " + vm.getUuid() + " did not reach target offering");
+ }
+ vm = resizedVm;
+ }
+
+ private void reconcileCapacity() {
+ if (!reconcileKubernetes) {
+ return;
+ }
+ kubernetesClusterNodeCapacityReconciler.onlineGuestResources(access);
+ kubernetesClusterNodeCapacityReconciler.verifyGuestResources(vm, targetOffering, before, access, scaleTimeoutTime);
+ if (!kubernetesClusterNodeCapacityReconciler.isKubernetesResourcesCurrent(before, targetOffering)) {
+ kubernetesClusterNodeCapacityReconciler.restartKubelet(vm, access, scaleTimeoutTime);
+ kubernetesClusterNodeCapacityReconciler.waitForKubernetesResources(kubernetesCluster, vm, targetOffering, before, access, scaleTimeoutTime);
+ }
+ kubernetesClusterNodeCapacityReconciler.restoreSchedulability(kubernetesCluster, vm, before, access, scaleTimeoutTime);
+ }
+
+ private void restoreAfterFailedResize() {
+ if (!reconcileKubernetes || !cksCordonCompleted || resizeSucceeded) {
+ return;
+ }
+ try {
+ kubernetesClusterNodeCapacityReconciler.restoreSchedulability(kubernetesCluster, vm, before, access, scaleTimeoutTime);
+ } catch (CloudRuntimeException cleanupException) {
+ logger.warn("Unable to restore schedulability for CKS node {} after a pre-resize failure", vm.getUuid(), cleanupException);
+ }
Review Comment:
If something other than a CloudRuntimeException is thrown, for example an NPE or a JSON parse error from nodeSnapshot, the cordon restore is skipped. The node then stays cordoned and no OperationFailed message is logged. Can we wrap those errors in NodeCapacityReconciliationException inside the reconciler, or restore the cordon in a finally, so we don't need a catch-all?
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterScaleWorker.java:
##########
@@ -376,24 +377,171 @@ private void scaleKubernetesClusterOffering(KubernetesClusterNodeType nodeType,
List<KubernetesClusterVmMapVO> vmList = kubernetesClusterVmMapDao.listByClusterIdAndVmType(kubernetesCluster.getId(), nodeType);
final long tobeScaledVMCount = Math.min(vmList.size(), size);
for (long i = 0; i < tobeScaledVMCount; i++) {
- KubernetesClusterVmMapVO vmMapVO = vmList.get((int) i);
- UserVmVO userVM = userVmDao.findById(vmMapVO.getVmId());
- boolean result = false;
- try {
- result = userVmManager.upgradeVirtualMachine(userVM.getId(), serviceOffering.getId(), new HashMap<String, String>());
- } catch (RuntimeException | ResourceUnavailableException | ManagementServerException | VirtualMachineMigrationException e) {
- logTransitStateAndThrow(Level.ERROR, String.format("Scaling Kubernetes cluster : %s failed, unable to scale cluster VM : %s due to %s", kubernetesCluster.getName(), userVM.getDisplayName(), e.getMessage()), kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed, e);
- }
- if (!result) {
- logTransitStateAndThrow(Level.WARN, String.format("Scaling Kubernetes cluster : %s failed, unable to scale cluster VM : %s", kubernetesCluster.getName(), userVM.getDisplayName()),kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
- }
+ new NodeOfferingResize(vmList.get((int) i), nodeType, serviceOffering).execute();
if (System.currentTimeMillis() > scaleTimeoutTime) {
logTransitStateAndThrow(Level.WARN, String.format("Scaling Kubernetes cluster : %s failed, scaling action timed out", kubernetesCluster.getName()),kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
}
}
kubernetesCluster = updateKubernetesClusterEntryForNodeType(null, nodeType, serviceOffering, updateNodeOffering, updateClusterOffering);
}
+ private final class NodeOfferingResize {
+ private final KubernetesClusterVmMapVO vmMap;
+ private final KubernetesClusterNodeType nodeType;
+ private final ServiceOffering targetOffering;
+ private final ServiceOffering oldOffering;
+ private UserVmVO vm;
+ private NodeAccess access;
+ private NodeCapacitySnapshot before;
+ private boolean reconcileKubernetes;
+ private boolean cksCordonCompleted;
+ private boolean resizeSucceeded;
+
+ private NodeOfferingResize(KubernetesClusterVmMapVO vmMap, KubernetesClusterNodeType nodeType, ServiceOffering targetOffering) {
+ this.vmMap = vmMap;
+ this.nodeType = nodeType;
+ this.targetOffering = targetOffering;
+ vm = userVmDao.findById(vmMap.getVmId());
+ if (vm == null) {
+ logTransitStateAndThrow(Level.ERROR, String.format("Scaling Kubernetes cluster : %s failed, unable to find cluster VM %s",
+ kubernetesCluster.getName(), vmMap.getVmId()), kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
+ }
+ oldOffering = serviceOfferingDao.findById(vm.getServiceOfferingId());
+ resizeSucceeded = Objects.equals(targetOffering.getId(), vm.getServiceOfferingId());
+ }
+
+ private void execute() {
+ validateLiveResize();
+ try {
+ prepareReconciliation();
+ resizeVm();
+ reconcileCapacity();
+ } catch (ResourceUnavailableException | ManagementServerException | VirtualMachineMigrationException | CloudRuntimeException e) {
+ restoreAfterFailedResize();
+ reportFailure(e);
+ }
+ }
+
+ private void validateLiveResize() {
+ if (KubernetesClusterNodeCapacityReconcilerImpl.capacityChanged(oldOffering, targetOffering)
+ && KubernetesCluster.State.Running == originalState && ETCD == nodeType
+ && KubernetesCluster.ClusterType.CloudManaged == kubernetesCluster.getClusterType()) {
+ logTransitStateAndThrow(Level.ERROR, String.format("Scaling Kubernetes cluster : %s cannot live-resize dedicated etcd VM %s; " +
+ "etcd health reconciliation is not implemented", kubernetesCluster.getName(), vm.getDisplayName()),
+ kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
+ }
Review Comment:
If we scale the control and etcd offerings together, either for the whole cluster or with a different offering per node type, the control nodes get scaled and the etcd check then fails. That leaves the cluster partially scaled. Can we move this check to `KubernetesClusterManagerImpl.validateServiceOfferingsForNodeTypesScale`and reject the request upfront, before any VM is touched?
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/utils/KubernetesClusterNodeCapacityReconcilerImpl.java:
##########
@@ -0,0 +1,232 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.kubernetes.cluster.utils;
+
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import com.cloud.kubernetes.cluster.KubernetesCluster;
+import com.cloud.kubernetes.cluster.KubernetesServiceHelper.KubernetesClusterNodeType;
+import com.cloud.offering.ServiceOffering;
+import com.cloud.uservm.UserVm;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.ssh.SshHelper;
+import com.google.gson.JsonArray;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParser;
+import org.apache.commons.lang3.StringUtils;
+
+/** SSH-backed implementation for a rolling CKS live-resize reconciliation. */
+public class KubernetesClusterNodeCapacityReconcilerImpl implements KubernetesClusterNodeCapacityReconciler {
+ // KVM QA observes 4,568,788 KiB in a 4,608 MiB guest after memory hotplug
+ // (about 146 MiB below the offering), so keep a small margin above that
+ // platform overhead while still rejecting materially undersized guests.
+ static final long MINIMUM_MEMORY_OVERHEAD_BYTES = 160L * 1024L * 1024L;
+ static final long KUBERNETES_MEMORY_REPORTING_TOLERANCE_BYTES = 16L * 1024L * 1024L;
+ private static final int COMMAND_TIMEOUT_MS = 30000;
+ private static final int POLL_INTERVAL_MS = 5000;
+ private static final Pattern NODE_NAME = Pattern.compile("[A-Za-z0-9][A-Za-z0-9.-]{0,252}");
+ private static final Pattern MEMORY_QUANTITY = Pattern.compile("([0-9]+)(Ki|Mi|Gi|Ti|K|M|G|T)?");
+ private static final String RESIZE_ANNOTATION = "cloudstack.apache.org/cks-live-resize";
+ static final String ONLINE_HOTPLUGGED_RESOURCES = "sudo sh -c 'for cpu in /sys/devices/system/cpu/cpu[0-9]*; do "
+ + "[ -e \"$cpu/online\" ] || continue; echo 1 > \"$cpu/online\"; done; "
+ + "for memory in /sys/devices/system/memory/memory[0-9]*; do "
+ + "[ -e \"$memory/online\" ] || continue; echo online > \"$memory/online\"; done'";
+
+ @Override
+ public boolean requiresKubeletRefresh(ServiceOffering oldOffering, ServiceOffering newOffering,
+ KubernetesClusterNodeType nodeType, KubernetesCluster.State originalState) {
+ return KubernetesCluster.State.Running == originalState
+ && (KubernetesClusterNodeType.WORKER == nodeType || KubernetesClusterNodeType.CONTROL == nodeType)
+ && capacityChanged(oldOffering, newOffering);
+ }
+
+ @Override
+ public NodeCapacitySnapshot captureBefore(KubernetesCluster cluster, UserVm vm, NodeAccess access) {
+ return nodeSnapshot(cluster, vm, access, guestResources(access));
+ }
+
+ @Override
+ public void cordonIfNeeded(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "=" + cluster.getUuid() + " --overwrite");
+ executeControl(access, "sudo /opt/bin/kubectl cordon " + node);
+ while (System.currentTimeMillis() < deadline) {
+ if (nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("CORDON", vm, "Kubernetes node did not become unschedulable");
Review Comment:
`KubernetesClusterUtil.uncordonKubernetesClusterNode` already runs `kubectl uncordon` with retries, and `KubernetesClusterUpgradeWorker` uses it. Can we reuse it here instead of a second copy? If a cordon helper is needed, we could add it to `KubernetesClusterUtil` next to it. The node-name lowercasing is also repeated.
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterScaleWorker.java:
##########
@@ -376,24 +377,171 @@ private void scaleKubernetesClusterOffering(KubernetesClusterNodeType nodeType,
List<KubernetesClusterVmMapVO> vmList = kubernetesClusterVmMapDao.listByClusterIdAndVmType(kubernetesCluster.getId(), nodeType);
final long tobeScaledVMCount = Math.min(vmList.size(), size);
for (long i = 0; i < tobeScaledVMCount; i++) {
- KubernetesClusterVmMapVO vmMapVO = vmList.get((int) i);
- UserVmVO userVM = userVmDao.findById(vmMapVO.getVmId());
- boolean result = false;
- try {
- result = userVmManager.upgradeVirtualMachine(userVM.getId(), serviceOffering.getId(), new HashMap<String, String>());
- } catch (RuntimeException | ResourceUnavailableException | ManagementServerException | VirtualMachineMigrationException e) {
- logTransitStateAndThrow(Level.ERROR, String.format("Scaling Kubernetes cluster : %s failed, unable to scale cluster VM : %s due to %s", kubernetesCluster.getName(), userVM.getDisplayName(), e.getMessage()), kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed, e);
- }
- if (!result) {
- logTransitStateAndThrow(Level.WARN, String.format("Scaling Kubernetes cluster : %s failed, unable to scale cluster VM : %s", kubernetesCluster.getName(), userVM.getDisplayName()),kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
- }
+ new NodeOfferingResize(vmList.get((int) i), nodeType, serviceOffering).execute();
if (System.currentTimeMillis() > scaleTimeoutTime) {
logTransitStateAndThrow(Level.WARN, String.format("Scaling Kubernetes cluster : %s failed, scaling action timed out", kubernetesCluster.getName()),kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
}
}
kubernetesCluster = updateKubernetesClusterEntryForNodeType(null, nodeType, serviceOffering, updateNodeOffering, updateClusterOffering);
}
+ private final class NodeOfferingResize {
+ private final KubernetesClusterVmMapVO vmMap;
+ private final KubernetesClusterNodeType nodeType;
+ private final ServiceOffering targetOffering;
+ private final ServiceOffering oldOffering;
+ private UserVmVO vm;
+ private NodeAccess access;
+ private NodeCapacitySnapshot before;
+ private boolean reconcileKubernetes;
+ private boolean cksCordonCompleted;
+ private boolean resizeSucceeded;
+
+ private NodeOfferingResize(KubernetesClusterVmMapVO vmMap, KubernetesClusterNodeType nodeType, ServiceOffering targetOffering) {
+ this.vmMap = vmMap;
+ this.nodeType = nodeType;
+ this.targetOffering = targetOffering;
+ vm = userVmDao.findById(vmMap.getVmId());
+ if (vm == null) {
+ logTransitStateAndThrow(Level.ERROR, String.format("Scaling Kubernetes cluster : %s failed, unable to find cluster VM %s",
+ kubernetesCluster.getName(), vmMap.getVmId()), kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
+ }
+ oldOffering = serviceOfferingDao.findById(vm.getServiceOfferingId());
+ resizeSucceeded = Objects.equals(targetOffering.getId(), vm.getServiceOfferingId());
+ }
+
+ private void execute() {
+ validateLiveResize();
+ try {
+ prepareReconciliation();
+ resizeVm();
+ reconcileCapacity();
+ } catch (ResourceUnavailableException | ManagementServerException | VirtualMachineMigrationException | CloudRuntimeException e) {
+ restoreAfterFailedResize();
+ reportFailure(e);
+ }
+ }
+
+ private void validateLiveResize() {
+ if (KubernetesClusterNodeCapacityReconcilerImpl.capacityChanged(oldOffering, targetOffering)
+ && KubernetesCluster.State.Running == originalState && ETCD == nodeType
+ && KubernetesCluster.ClusterType.CloudManaged == kubernetesCluster.getClusterType()) {
+ logTransitStateAndThrow(Level.ERROR, String.format("Scaling Kubernetes cluster : %s cannot live-resize dedicated etcd VM %s; " +
+ "etcd health reconciliation is not implemented", kubernetesCluster.getName(), vm.getDisplayName()),
+ kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
+ }
+ }
+
+ private void prepareReconciliation() {
+ reconcileKubernetes = shouldReconcileNodeCapacity(vmMap, vm, oldOffering, targetOffering, nodeType);
+ if (reconcileKubernetes || isReconciliationRetryCandidate(vmMap, vm, targetOffering, nodeType)) {
+ access = resolveNodeAccess(vm);
+ before = kubernetesClusterNodeCapacityReconciler.captureBefore(kubernetesCluster, vm, access);
+ reconcileKubernetes = reconcileKubernetes || before.isCloudStackResizeCordon();
+ }
+ if (reconcileKubernetes) {
+ kubernetesClusterNodeCapacityReconciler.cordonIfNeeded(kubernetesCluster, vm, before, access, scaleTimeoutTime);
+ cksCordonCompleted = !before.isUnschedulable() || before.isCloudStackResizeCordon();
+ }
+ }
+
+ private void resizeVm() throws ResourceUnavailableException, ManagementServerException, VirtualMachineMigrationException {
+ if (resizeSucceeded) {
+ return;
+ }
+ if (!userVmManager.upgradeVirtualMachine(vm.getId(), targetOffering.getId(), new HashMap<>())) {
+ throw new CloudRuntimeException("Unable to scale cluster VM " + vm.getDisplayName());
+ }
+ resizeSucceeded = true;
+ UserVmVO resizedVm = userVmDao.findById(vm.getId());
+ if (resizedVm == null || !Objects.equals(targetOffering.getId(), resizedVm.getServiceOfferingId())) {
+ throw new CloudRuntimeException("VM " + vm.getUuid() + " did not reach target offering");
+ }
+ vm = resizedVm;
+ }
+
+ private void reconcileCapacity() {
+ if (!reconcileKubernetes) {
+ return;
+ }
+ kubernetesClusterNodeCapacityReconciler.onlineGuestResources(access);
+ kubernetesClusterNodeCapacityReconciler.verifyGuestResources(vm, targetOffering, before, access, scaleTimeoutTime);
+ if (!kubernetesClusterNodeCapacityReconciler.isKubernetesResourcesCurrent(before, targetOffering)) {
+ kubernetesClusterNodeCapacityReconciler.restartKubelet(vm, access, scaleTimeoutTime);
+ kubernetesClusterNodeCapacityReconciler.waitForKubernetesResources(kubernetesCluster, vm, targetOffering, before, access, scaleTimeoutTime);
+ }
+ kubernetesClusterNodeCapacityReconciler.restoreSchedulability(kubernetesCluster, vm, before, access, scaleTimeoutTime);
+ }
+
+ private void restoreAfterFailedResize() {
+ if (!reconcileKubernetes || !cksCordonCompleted || resizeSucceeded) {
+ return;
+ }
+ try {
+ kubernetesClusterNodeCapacityReconciler.restoreSchedulability(kubernetesCluster, vm, before, access, scaleTimeoutTime);
+ } catch (CloudRuntimeException cleanupException) {
+ logger.warn("Unable to restore schedulability for CKS node {} after a pre-resize failure", vm.getUuid(), cleanupException);
+ }
+ }
+
+ private void reportFailure(Exception cause) {
+ String recovery = resizeSucceeded
+ ? "The node remains cordoned; check kubelet on the VM and retry the CKS scale operation."
+ : "The VM was not resized; CKS attempted to restore its temporary cordon.";
+ String message = String.format("Scaling Kubernetes cluster %s (UUID %s) failed for %s node VM %s (UUID %s), " +
+ "offering %s to %s: %s %s", kubernetesCluster.getName(), kubernetesCluster.getUuid(), nodeType,
+ vm.getDisplayName(), vm.getUuid(), oldOffering == null ? null : oldOffering.getId(),
+ targetOffering.getId(), cause.getMessage(), recovery);
+ logTransitStateAndThrow(Level.ERROR, message, kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed, cause);
+ }
+ }
+
+ private NodeAccess resolveNodeAccess(UserVm userVM) {
+ Pair<String, Integer> controlAccess = getKubernetesClusterServerIpSshPort(null);
+ if (StringUtils.isBlank(controlAccess.first())) {
+ throw new CloudRuntimeException(String.format("Unable to resolve control-plane SSH for Kubernetes cluster %s", kubernetesCluster.getUuid()));
+ }
+ if (manager.isDirectAccess(network)) {
+ if (StringUtils.isBlank(userVM.getPrivateIpAddress())) {
+ throw new CloudRuntimeException(String.format("Unable to resolve private SSH address for VM %s", userVM.getUuid()));
+ }
+ return new NodeAccess(controlAccess.first(), controlAccess.second(), userVM.getPrivateIpAddress(), DEFAULT_SSH_PORT,
+ getControlNodeLoginUser(), getManagementServerSshPublicKeyFile());
+ }
+ PortForwardingRuleVO sshRule = portForwardingRulesDao.listByVm(userVM.getId()).stream()
+ .filter(rule -> rule.getDestinationPortStart() == DEFAULT_SSH_PORT)
+ .filter(rule -> !FirewallRule.State.Revoke.equals(rule.getState()))
+ .findFirst().orElse(null);
+ if (sshRule == null) {
+ throw new CloudRuntimeException(String.format("Unable to resolve SSH port-forwarding rule for VM %s", userVM.getUuid()));
+ }
+ IpAddress targetPublicIp = network.getVpcId() == null ? getNetworkSourceNatIp(network) : getVpcTierKubernetesPublicIp(network);
+ if (targetPublicIp == null) {
+ throw new CloudRuntimeException(String.format("Unable to resolve target-node public IP for VM %s", userVM.getUuid()));
+ }
+ return new NodeAccess(controlAccess.first(), controlAccess.second(), targetPublicIp.getAddress().addr(), sshRule.getSourcePortStart(),
+ getControlNodeLoginUser(), getManagementServerSshPublicKeyFile());
+ }
+
+ protected boolean shouldReconcileNodeCapacity(KubernetesClusterVmMapVO vmMapVO, UserVm userVM,
+ ServiceOffering oldOffering, ServiceOffering targetOffering,
+ KubernetesClusterNodeType nodeType) {
+ return !vmMapVO.isExternalNode()
+ && KubernetesCluster.ClusterType.CloudManaged == kubernetesCluster.getClusterType()
+ && Hypervisor.HypervisorType.KVM == userVM.getHypervisorType()
+ && kubernetesClusterNodeCapacityReconciler.requiresKubeletRefresh(oldOffering, targetOffering, nodeType, originalState);
+ }
+
+ protected boolean isReconciliationRetryCandidate(KubernetesClusterVmMapVO vmMapVO, UserVm userVM,
+ ServiceOffering targetOffering, KubernetesClusterNodeType nodeType) {
+ return !vmMapVO.isExternalNode()
+ && KubernetesCluster.ClusterType.CloudManaged == kubernetesCluster.getClusterType()
+ && Hypervisor.HypervisorType.KVM == userVM.getHypervisorType()
+ && (WORKER == nodeType || CONTROL == nodeType)
+ && KubernetesCluster.State.Running == originalState
Review Comment:
These 2 methods look quite similar, can we extract one of them?
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterScaleWorker.java:
##########
@@ -376,24 +377,171 @@ private void scaleKubernetesClusterOffering(KubernetesClusterNodeType nodeType,
List<KubernetesClusterVmMapVO> vmList = kubernetesClusterVmMapDao.listByClusterIdAndVmType(kubernetesCluster.getId(), nodeType);
final long tobeScaledVMCount = Math.min(vmList.size(), size);
for (long i = 0; i < tobeScaledVMCount; i++) {
- KubernetesClusterVmMapVO vmMapVO = vmList.get((int) i);
- UserVmVO userVM = userVmDao.findById(vmMapVO.getVmId());
- boolean result = false;
- try {
- result = userVmManager.upgradeVirtualMachine(userVM.getId(), serviceOffering.getId(), new HashMap<String, String>());
- } catch (RuntimeException | ResourceUnavailableException | ManagementServerException | VirtualMachineMigrationException e) {
- logTransitStateAndThrow(Level.ERROR, String.format("Scaling Kubernetes cluster : %s failed, unable to scale cluster VM : %s due to %s", kubernetesCluster.getName(), userVM.getDisplayName(), e.getMessage()), kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed, e);
- }
- if (!result) {
- logTransitStateAndThrow(Level.WARN, String.format("Scaling Kubernetes cluster : %s failed, unable to scale cluster VM : %s", kubernetesCluster.getName(), userVM.getDisplayName()),kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
- }
+ new NodeOfferingResize(vmList.get((int) i), nodeType, serviceOffering).execute();
if (System.currentTimeMillis() > scaleTimeoutTime) {
logTransitStateAndThrow(Level.WARN, String.format("Scaling Kubernetes cluster : %s failed, scaling action timed out", kubernetesCluster.getName()),kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
}
}
kubernetesCluster = updateKubernetesClusterEntryForNodeType(null, nodeType, serviceOffering, updateNodeOffering, updateClusterOffering);
}
+ private final class NodeOfferingResize {
Review Comment:
Can we move NodeOfferingResize into a separate class?
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterScaleWorker.java:
##########
@@ -229,32 +236,36 @@ private KubernetesClusterVO updateKubernetesClusterEntryForNodeType(final Long n
}
protected Pair<Long, Long> calculateNewClusterCountAndCapacity(Long newWorkerSize, KubernetesClusterNodeType nodeType, ServiceOffering serviceOffering) {
- long cores;
- long memory;
- long totalClusterSize = newWorkerSize == null ? kubernetesCluster.getTotalNodeCount() : (newWorkerSize + kubernetesCluster.getControlNodeCount() + kubernetesCluster.getEtcdNodeCount());
-
- if (nodeType == DEFAULT) {
- cores = serviceOffering.getCpu() * totalClusterSize;
- memory = serviceOffering.getRamSize() * totalClusterSize;
- } else {
- long nodeCount = getNodeCountForType(nodeType, kubernetesCluster);
- Long existingOfferingId = getExistingOfferingIdForNodeType(nodeType, kubernetesCluster);
- if (existingOfferingId == null) {
- existingOfferingId = serviceOffering.getId();
- }
- ServiceOfferingVO previousOffering = serviceOfferingDao.findById(existingOfferingId);
- Pair<Long, Long> previousNodesCapacity = calculateNodesCapacity(previousOffering, nodeCount);
- if (WORKER == nodeType) {
- nodeCount = newWorkerSize == null ? kubernetesCluster.getNodeCount() : newWorkerSize;
+ long cores = 0;
+ long memory = 0;
+ // VM offerings have already changed here. Recompute the total from the
+ // configured node types instead of subtracting a supposedly old VM offering.
+ for (KubernetesClusterNodeType type : List.of(CONTROL, WORKER, ETCD)) {
+ long nodeCount = WORKER == type && newWorkerSize != null ? newWorkerSize : getNodeCountForType(type, kubernetesCluster);
+ if (nodeCount == 0) {
+ continue;
}
- Pair<Long, Long> newNodesCapacity = calculateNodesCapacity(serviceOffering, nodeCount);
- Pair<Long, Long> newClusterCapacity = calculateClusterNewCapacity(kubernetesCluster, previousNodesCapacity, newNodesCapacity);
- cores = newClusterCapacity.first();
- memory = newClusterCapacity.second();
+ ServiceOffering offering = nodeType == DEFAULT || nodeType == type ? serviceOffering :
+ serviceOfferingDao.findByIdIncludingRemoved(getConfiguredOfferingIdForNodeType(type));
Review Comment:
can we reuse `KubernetesClusterActionWorker.getServiceOfferingForNodeTypeOnCluster(type, kubernetesCluster)` instead?
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterScaleWorker.java:
##########
@@ -229,32 +236,36 @@ private KubernetesClusterVO updateKubernetesClusterEntryForNodeType(final Long n
}
protected Pair<Long, Long> calculateNewClusterCountAndCapacity(Long newWorkerSize, KubernetesClusterNodeType nodeType, ServiceOffering serviceOffering) {
- long cores;
- long memory;
- long totalClusterSize = newWorkerSize == null ? kubernetesCluster.getTotalNodeCount() : (newWorkerSize + kubernetesCluster.getControlNodeCount() + kubernetesCluster.getEtcdNodeCount());
-
- if (nodeType == DEFAULT) {
- cores = serviceOffering.getCpu() * totalClusterSize;
- memory = serviceOffering.getRamSize() * totalClusterSize;
- } else {
- long nodeCount = getNodeCountForType(nodeType, kubernetesCluster);
- Long existingOfferingId = getExistingOfferingIdForNodeType(nodeType, kubernetesCluster);
- if (existingOfferingId == null) {
- existingOfferingId = serviceOffering.getId();
- }
- ServiceOfferingVO previousOffering = serviceOfferingDao.findById(existingOfferingId);
- Pair<Long, Long> previousNodesCapacity = calculateNodesCapacity(previousOffering, nodeCount);
- if (WORKER == nodeType) {
- nodeCount = newWorkerSize == null ? kubernetesCluster.getNodeCount() : newWorkerSize;
+ long cores = 0;
+ long memory = 0;
+ // VM offerings have already changed here. Recompute the total from the
+ // configured node types instead of subtracting a supposedly old VM offering.
+ for (KubernetesClusterNodeType type : List.of(CONTROL, WORKER, ETCD)) {
+ long nodeCount = WORKER == type && newWorkerSize != null ? newWorkerSize : getNodeCountForType(type, kubernetesCluster);
+ if (nodeCount == 0) {
+ continue;
}
- Pair<Long, Long> newNodesCapacity = calculateNodesCapacity(serviceOffering, nodeCount);
- Pair<Long, Long> newClusterCapacity = calculateClusterNewCapacity(kubernetesCluster, previousNodesCapacity, newNodesCapacity);
- cores = newClusterCapacity.first();
- memory = newClusterCapacity.second();
+ ServiceOffering offering = nodeType == DEFAULT || nodeType == type ? serviceOffering :
+ serviceOfferingDao.findByIdIncludingRemoved(getConfiguredOfferingIdForNodeType(type));
+ Pair<Long, Long> capacity = calculateNodesCapacity(offering, nodeCount);
+ cores += capacity.first();
+ memory += capacity.second();
}
return new Pair<>(cores, memory);
}
+ private Long getConfiguredOfferingIdForNodeType(KubernetesClusterNodeType nodeType) {
+ Long offeringId = null;
+ if (WORKER == nodeType) {
+ offeringId = kubernetesCluster.getWorkerNodeServiceOfferingId();
+ } else if (CONTROL == nodeType) {
+ offeringId = kubernetesCluster.getControlNodeServiceOfferingId();
+ } else if (ETCD == nodeType) {
+ offeringId = kubernetesCluster.getEtcdNodeServiceOfferingId();
+ }
+ return offeringId == null ? kubernetesCluster.getServiceOfferingId() : offeringId;
+ }
Review Comment:
`getConfiguredOfferingIdForNodeType` does the same thing as `getServiceOfferingForNodeTypeOnCluster` in `KubernetesClusterActionWorker` (per-type offering, falling back to the cluster offering). Can we reuse that instead of adding another copy? Also, if both the node-type and cluster offering ids are null, this returns null and the callers then call `findById(null)`.
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/utils/KubernetesClusterNodeCapacityReconcilerImpl.java:
##########
@@ -0,0 +1,232 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.kubernetes.cluster.utils;
+
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import com.cloud.kubernetes.cluster.KubernetesCluster;
+import com.cloud.kubernetes.cluster.KubernetesServiceHelper.KubernetesClusterNodeType;
+import com.cloud.offering.ServiceOffering;
+import com.cloud.uservm.UserVm;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.ssh.SshHelper;
+import com.google.gson.JsonArray;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParser;
+import org.apache.commons.lang3.StringUtils;
+
+/** SSH-backed implementation for a rolling CKS live-resize reconciliation. */
+public class KubernetesClusterNodeCapacityReconcilerImpl implements KubernetesClusterNodeCapacityReconciler {
+ // KVM QA observes 4,568,788 KiB in a 4,608 MiB guest after memory hotplug
+ // (about 146 MiB below the offering), so keep a small margin above that
+ // platform overhead while still rejecting materially undersized guests.
+ static final long MINIMUM_MEMORY_OVERHEAD_BYTES = 160L * 1024L * 1024L;
+ static final long KUBERNETES_MEMORY_REPORTING_TOLERANCE_BYTES = 16L * 1024L * 1024L;
+ private static final int COMMAND_TIMEOUT_MS = 30000;
+ private static final int POLL_INTERVAL_MS = 5000;
+ private static final Pattern NODE_NAME = Pattern.compile("[A-Za-z0-9][A-Za-z0-9.-]{0,252}");
+ private static final Pattern MEMORY_QUANTITY = Pattern.compile("([0-9]+)(Ki|Mi|Gi|Ti|K|M|G|T)?");
+ private static final String RESIZE_ANNOTATION = "cloudstack.apache.org/cks-live-resize";
+ static final String ONLINE_HOTPLUGGED_RESOURCES = "sudo sh -c 'for cpu in /sys/devices/system/cpu/cpu[0-9]*; do "
+ + "[ -e \"$cpu/online\" ] || continue; echo 1 > \"$cpu/online\"; done; "
+ + "for memory in /sys/devices/system/memory/memory[0-9]*; do "
+ + "[ -e \"$memory/online\" ] || continue; echo online > \"$memory/online\"; done'";
+
+ @Override
+ public boolean requiresKubeletRefresh(ServiceOffering oldOffering, ServiceOffering newOffering,
+ KubernetesClusterNodeType nodeType, KubernetesCluster.State originalState) {
+ return KubernetesCluster.State.Running == originalState
+ && (KubernetesClusterNodeType.WORKER == nodeType || KubernetesClusterNodeType.CONTROL == nodeType)
+ && capacityChanged(oldOffering, newOffering);
+ }
+
+ @Override
+ public NodeCapacitySnapshot captureBefore(KubernetesCluster cluster, UserVm vm, NodeAccess access) {
+ return nodeSnapshot(cluster, vm, access, guestResources(access));
+ }
+
+ @Override
+ public void cordonIfNeeded(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "=" + cluster.getUuid() + " --overwrite");
+ executeControl(access, "sudo /opt/bin/kubectl cordon " + node);
+ while (System.currentTimeMillis() < deadline) {
+ if (nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("CORDON", vm, "Kubernetes node did not become unschedulable");
+ }
+
+ @Override
+ public void onlineGuestResources(NodeAccess access) {
+ executeNode(access, ONLINE_HOTPLUGGED_RESOURCES);
+ }
+
+ @Override
+ public void verifyGuestResources(UserVm vm, ServiceOffering target, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ if (guestMatchesTarget(guestResources(access), target)) return;
+ sleep();
+ }
+ throw failure("GUEST_VERIFY", vm, "guest CPU or memory did not reach the target offering");
+ }
+
+ @Override
+ public boolean isKubernetesResourcesCurrent(NodeCapacitySnapshot snapshot, ServiceOffering target) {
+ GuestResources guest = new GuestResources(snapshot.getGuestOnlineCpuCount(), snapshot.getGuestMemoryKiB());
+ return guestMatchesTarget(guest, target) && kubernetesMatchesGuest(snapshot, snapshot);
Review Comment:
`kubernetesMatchesGuest(snapshot, snapshot)` compares the snapshot with itself, so its "allocatable grew" checks always pass. Can we fix that? Also, can we rename this method to something like `isKubernetesNodeAlreadyResized` ?
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/utils/KubernetesClusterNodeCapacityReconcilerImpl.java:
##########
@@ -0,0 +1,232 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.kubernetes.cluster.utils;
+
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import com.cloud.kubernetes.cluster.KubernetesCluster;
+import com.cloud.kubernetes.cluster.KubernetesServiceHelper.KubernetesClusterNodeType;
+import com.cloud.offering.ServiceOffering;
+import com.cloud.uservm.UserVm;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.ssh.SshHelper;
+import com.google.gson.JsonArray;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParser;
+import org.apache.commons.lang3.StringUtils;
+
+/** SSH-backed implementation for a rolling CKS live-resize reconciliation. */
+public class KubernetesClusterNodeCapacityReconcilerImpl implements KubernetesClusterNodeCapacityReconciler {
+ // KVM QA observes 4,568,788 KiB in a 4,608 MiB guest after memory hotplug
+ // (about 146 MiB below the offering), so keep a small margin above that
+ // platform overhead while still rejecting materially undersized guests.
+ static final long MINIMUM_MEMORY_OVERHEAD_BYTES = 160L * 1024L * 1024L;
+ static final long KUBERNETES_MEMORY_REPORTING_TOLERANCE_BYTES = 16L * 1024L * 1024L;
+ private static final int COMMAND_TIMEOUT_MS = 30000;
+ private static final int POLL_INTERVAL_MS = 5000;
+ private static final Pattern NODE_NAME = Pattern.compile("[A-Za-z0-9][A-Za-z0-9.-]{0,252}");
+ private static final Pattern MEMORY_QUANTITY = Pattern.compile("([0-9]+)(Ki|Mi|Gi|Ti|K|M|G|T)?");
+ private static final String RESIZE_ANNOTATION = "cloudstack.apache.org/cks-live-resize";
+ static final String ONLINE_HOTPLUGGED_RESOURCES = "sudo sh -c 'for cpu in /sys/devices/system/cpu/cpu[0-9]*; do "
+ + "[ -e \"$cpu/online\" ] || continue; echo 1 > \"$cpu/online\"; done; "
+ + "for memory in /sys/devices/system/memory/memory[0-9]*; do "
+ + "[ -e \"$memory/online\" ] || continue; echo online > \"$memory/online\"; done'";
+
+ @Override
+ public boolean requiresKubeletRefresh(ServiceOffering oldOffering, ServiceOffering newOffering,
+ KubernetesClusterNodeType nodeType, KubernetesCluster.State originalState) {
+ return KubernetesCluster.State.Running == originalState
+ && (KubernetesClusterNodeType.WORKER == nodeType || KubernetesClusterNodeType.CONTROL == nodeType)
+ && capacityChanged(oldOffering, newOffering);
+ }
+
+ @Override
+ public NodeCapacitySnapshot captureBefore(KubernetesCluster cluster, UserVm vm, NodeAccess access) {
+ return nodeSnapshot(cluster, vm, access, guestResources(access));
+ }
+
+ @Override
+ public void cordonIfNeeded(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "=" + cluster.getUuid() + " --overwrite");
+ executeControl(access, "sudo /opt/bin/kubectl cordon " + node);
+ while (System.currentTimeMillis() < deadline) {
Review Comment:
This loop is repeated five times in this class, in `cordonIfNeeded`, `verifyGuestResources`, `restartKubelet`, `waitForKubernetesResources` and `restoreSchedulability`. Each one checks a condition, sleeps and throws a phase-specific failure when the deadline passes. Can we extract one helper, for example pollUntil(deadline, condition, phase, vm, message), and use it in all five?
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/utils/KubernetesClusterNodeCapacityReconcilerImpl.java:
##########
@@ -0,0 +1,232 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.kubernetes.cluster.utils;
+
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import com.cloud.kubernetes.cluster.KubernetesCluster;
+import com.cloud.kubernetes.cluster.KubernetesServiceHelper.KubernetesClusterNodeType;
+import com.cloud.offering.ServiceOffering;
+import com.cloud.uservm.UserVm;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.ssh.SshHelper;
+import com.google.gson.JsonArray;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParser;
+import org.apache.commons.lang3.StringUtils;
+
+/** SSH-backed implementation for a rolling CKS live-resize reconciliation. */
+public class KubernetesClusterNodeCapacityReconcilerImpl implements KubernetesClusterNodeCapacityReconciler {
+ // KVM QA observes 4,568,788 KiB in a 4,608 MiB guest after memory hotplug
+ // (about 146 MiB below the offering), so keep a small margin above that
+ // platform overhead while still rejecting materially undersized guests.
+ static final long MINIMUM_MEMORY_OVERHEAD_BYTES = 160L * 1024L * 1024L;
+ static final long KUBERNETES_MEMORY_REPORTING_TOLERANCE_BYTES = 16L * 1024L * 1024L;
+ private static final int COMMAND_TIMEOUT_MS = 30000;
+ private static final int POLL_INTERVAL_MS = 5000;
+ private static final Pattern NODE_NAME = Pattern.compile("[A-Za-z0-9][A-Za-z0-9.-]{0,252}");
+ private static final Pattern MEMORY_QUANTITY = Pattern.compile("([0-9]+)(Ki|Mi|Gi|Ti|K|M|G|T)?");
+ private static final String RESIZE_ANNOTATION = "cloudstack.apache.org/cks-live-resize";
+ static final String ONLINE_HOTPLUGGED_RESOURCES = "sudo sh -c 'for cpu in /sys/devices/system/cpu/cpu[0-9]*; do "
+ + "[ -e \"$cpu/online\" ] || continue; echo 1 > \"$cpu/online\"; done; "
+ + "for memory in /sys/devices/system/memory/memory[0-9]*; do "
+ + "[ -e \"$memory/online\" ] || continue; echo online > \"$memory/online\"; done'";
+
+ @Override
+ public boolean requiresKubeletRefresh(ServiceOffering oldOffering, ServiceOffering newOffering,
+ KubernetesClusterNodeType nodeType, KubernetesCluster.State originalState) {
+ return KubernetesCluster.State.Running == originalState
+ && (KubernetesClusterNodeType.WORKER == nodeType || KubernetesClusterNodeType.CONTROL == nodeType)
+ && capacityChanged(oldOffering, newOffering);
+ }
+
+ @Override
+ public NodeCapacitySnapshot captureBefore(KubernetesCluster cluster, UserVm vm, NodeAccess access) {
+ return nodeSnapshot(cluster, vm, access, guestResources(access));
+ }
+
+ @Override
+ public void cordonIfNeeded(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "=" + cluster.getUuid() + " --overwrite");
+ executeControl(access, "sudo /opt/bin/kubectl cordon " + node);
+ while (System.currentTimeMillis() < deadline) {
+ if (nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("CORDON", vm, "Kubernetes node did not become unschedulable");
+ }
+
+ @Override
+ public void onlineGuestResources(NodeAccess access) {
+ executeNode(access, ONLINE_HOTPLUGGED_RESOURCES);
+ }
+
+ @Override
+ public void verifyGuestResources(UserVm vm, ServiceOffering target, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ if (guestMatchesTarget(guestResources(access), target)) return;
+ sleep();
+ }
+ throw failure("GUEST_VERIFY", vm, "guest CPU or memory did not reach the target offering");
+ }
+
+ @Override
+ public boolean isKubernetesResourcesCurrent(NodeCapacitySnapshot snapshot, ServiceOffering target) {
+ GuestResources guest = new GuestResources(snapshot.getGuestOnlineCpuCount(), snapshot.getGuestMemoryKiB());
+ return guestMatchesTarget(guest, target) && kubernetesMatchesGuest(snapshot, snapshot);
+ }
+
+ @Override
+ public void restartKubelet(UserVm vm, NodeAccess access, long deadline) {
+ executeNode(access, "sudo systemctl restart kubelet");
+ while (System.currentTimeMillis() < deadline) {
+ if ("active".equals(executeNode(access, "sudo systemctl is-active kubelet").trim())) return;
+ sleep();
+ }
+ throw failure("KUBELET_RESTART", vm, "kubelet did not become active");
+ }
+
+ @Override
+ public NodeCapacitySnapshot waitForKubernetesResources(KubernetesCluster cluster, UserVm vm, ServiceOffering target,
+ NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ GuestResources guest = guestResources(access);
+ NodeCapacitySnapshot observed = nodeSnapshot(cluster, vm, access, guest);
+ if (guestMatchesTarget(guest, target) && kubernetesMatchesGuest(observed, before)) return observed;
+ sleep();
+ }
+ throw failure("CAPACITY_VERIFY", vm, "Kubernetes capacity did not match the resized guest");
+ }
+
+ @Override
+ public void restoreSchedulability(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl uncordon " + node);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "-");
+ while (System.currentTimeMillis() < deadline) {
+ if (!nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("RESTORE_SCHEDULABILITY", vm, "Kubernetes node remained unschedulable");
+ }
+
+ public static boolean capacityChanged(ServiceOffering oldOffering, ServiceOffering targetOffering) {
+ return oldOffering != null && targetOffering != null
+ && (!Objects.equals(oldOffering.getCpu(), targetOffering.getCpu())
+ || !Objects.equals(oldOffering.getRamSize(), targetOffering.getRamSize()));
+ }
+
+ static long parseCpuMillis(String value) { return value.endsWith("m") ? Long.parseLong(value.substring(0, value.length() - 1)) : Long.parseLong(value) * 1000L; }
+ static long parseMemoryBytes(String value) {
+ Matcher matcher = MEMORY_QUANTITY.matcher(value);
+ if (!matcher.matches()) throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ long number = Long.parseLong(matcher.group(1)); String unit = matcher.group(2);
+ if (unit == null) return number;
+ switch (unit) {
+ case "Ki": return number * 1024L; case "Mi": return number * 1024L * 1024L; case "Gi": return number * 1024L * 1024L * 1024L;
+ case "Ti": return number * 1024L * 1024L * 1024L * 1024L; case "K": return number * 1000L; case "M": return number * 1000L * 1000L;
+ case "G": return number * 1000L * 1000L * 1000L; case "T": return number * 1000L * 1000L * 1000L * 1000L;
+ default: throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ }
+ }
+
+ private NodeCapacitySnapshot nodeSnapshot(KubernetesCluster cluster, UserVm vm, NodeAccess access, GuestResources guest) {
+ JsonObject node = new JsonParser().parse(executeControl(access, "sudo /opt/bin/kubectl get node " + nodeName(vm) + " -o json")).getAsJsonObject();
+ JsonObject annotations = node.getAsJsonObject("metadata").has("annotations") ? node.getAsJsonObject("metadata").getAsJsonObject("annotations") : null;
+ boolean cksCordon = annotations != null && annotations.has(RESIZE_ANNOTATION) && cluster.getUuid().equals(annotations.get(RESIZE_ANNOTATION).getAsString());
+ JsonObject spec = node.has("spec") ? node.getAsJsonObject("spec") : new JsonObject(); JsonObject status = node.getAsJsonObject("status");
+ JsonObject capacity = status.getAsJsonObject("capacity"); JsonObject allocatable = status.getAsJsonObject("allocatable");
+ return new NodeCapacitySnapshot(spec.has("unschedulable") && spec.get("unschedulable").getAsBoolean(), cksCordon,
+ guest == null ? 0 : guest.cpu, guest == null ? 0 : guest.memoryKiB, parseCpuMillis(capacity.get("cpu").getAsString()),
+ parseMemoryBytes(capacity.get("memory").getAsString()), parseCpuMillis(allocatable.get("cpu").getAsString()),
+ parseMemoryBytes(allocatable.get("memory").getAsString()), isReady(status));
+ }
+
+ private boolean isReady(JsonObject status) {
+ JsonArray conditions = status.getAsJsonArray("conditions");
+ for (JsonElement condition : conditions) { JsonObject item = condition.getAsJsonObject(); if ("Ready".equals(item.get("type").getAsString()) && "True".equals(item.get("status").getAsString())) return true; }
+ return false;
+ }
+
+ private GuestResources guestResources(NodeAccess access) {
+ String[] values = executeNode(access, "getconf _NPROCESSORS_ONLN; awk '/^MemTotal:/ {print $2}' /proc/meminfo").trim().split("\\s+");
+ if (values.length != 2) throw new CloudRuntimeException("Unable to read guest CPU and memory");
+ return new GuestResources(Long.parseLong(values[0]), Long.parseLong(values[1]));
+ }
+
+ private boolean guestMatchesTarget(GuestResources observed, ServiceOffering target) {
Review Comment:
nit: for the sake of understanding (in future as well) can we add a small comment for `guestMatchesTarget` and `kubernetesMatchesGuest` on what it does. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "ACSHomeBot (via GitHub)" <gi...@apache.org>.
ACSHomeBot commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6470138868
Accepted: `package kvm G` for `c48bd344eeb1`.
Planned: **RPM + DEB**, **KVM SystemVM**.
<!-- acshomebot-package-state:20261028T132312Z-adeb3d74 -->
Status: **Queued** for **worker G**. Jobs ahead: **0**.
<!-- /acshomebot-package-state:20261028T132312Z-adeb3d74 -->
<!-- acshomebot-package-status:20261028T132312Z-adeb3d74 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "andrijapanicsb (via GitHub)" <gi...@apache.org>.
andrijapanicsb commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6470127936
@ACSHomeBot package G
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "DaanHoogland (via GitHub) via commits" <co...@cloudstack.apache.org>.
DaanHoogland commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6636334712
sus build failed:
```
02:52:41 [ERROR] Failures:
02:52:41 [ERROR] MetalinkDirectTemplateDownloaderTest.testCheckUrlMetalink:33
```
looks like a generic test so rebuilding in the background
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "andrijapanicsb (via GitHub)" <gi...@apache.org>.
andrijapanicsb commented on code in PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#discussion_r4394977644
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/utils/KubernetesClusterNodeCapacityReconcilerImpl.java:
##########
@@ -0,0 +1,232 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.kubernetes.cluster.utils;
+
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import com.cloud.kubernetes.cluster.KubernetesCluster;
+import com.cloud.kubernetes.cluster.KubernetesServiceHelper.KubernetesClusterNodeType;
+import com.cloud.offering.ServiceOffering;
+import com.cloud.uservm.UserVm;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.ssh.SshHelper;
+import com.google.gson.JsonArray;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParser;
+import org.apache.commons.lang3.StringUtils;
+
+/** SSH-backed implementation for a rolling CKS live-resize reconciliation. */
+public class KubernetesClusterNodeCapacityReconcilerImpl implements KubernetesClusterNodeCapacityReconciler {
+ // KVM QA observes 4,568,788 KiB in a 4,608 MiB guest after memory hotplug
+ // (about 146 MiB below the offering), so keep a small margin above that
+ // platform overhead while still rejecting materially undersized guests.
+ static final long MINIMUM_MEMORY_OVERHEAD_BYTES = 160L * 1024L * 1024L;
+ static final long KUBERNETES_MEMORY_REPORTING_TOLERANCE_BYTES = 16L * 1024L * 1024L;
+ private static final int COMMAND_TIMEOUT_MS = 30000;
+ private static final int POLL_INTERVAL_MS = 5000;
+ private static final Pattern NODE_NAME = Pattern.compile("[A-Za-z0-9][A-Za-z0-9.-]{0,252}");
+ private static final Pattern MEMORY_QUANTITY = Pattern.compile("([0-9]+)(Ki|Mi|Gi|Ti|K|M|G|T)?");
+ private static final String RESIZE_ANNOTATION = "cloudstack.apache.org/cks-live-resize";
+ static final String ONLINE_HOTPLUGGED_RESOURCES = "sudo sh -c 'for cpu in /sys/devices/system/cpu/cpu[0-9]*; do "
+ + "[ -e \"$cpu/online\" ] || continue; echo 1 > \"$cpu/online\"; done; "
+ + "for memory in /sys/devices/system/memory/memory[0-9]*; do "
+ + "[ -e \"$memory/online\" ] || continue; echo online > \"$memory/online\"; done'";
+
+ @Override
+ public boolean requiresKubeletRefresh(ServiceOffering oldOffering, ServiceOffering newOffering,
+ KubernetesClusterNodeType nodeType, KubernetesCluster.State originalState) {
+ return KubernetesCluster.State.Running == originalState
+ && (KubernetesClusterNodeType.WORKER == nodeType || KubernetesClusterNodeType.CONTROL == nodeType)
+ && capacityChanged(oldOffering, newOffering);
+ }
+
+ @Override
+ public NodeCapacitySnapshot captureBefore(KubernetesCluster cluster, UserVm vm, NodeAccess access) {
+ return nodeSnapshot(cluster, vm, access, guestResources(access));
+ }
+
+ @Override
+ public void cordonIfNeeded(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "=" + cluster.getUuid() + " --overwrite");
+ executeControl(access, "sudo /opt/bin/kubectl cordon " + node);
+ while (System.currentTimeMillis() < deadline) {
+ if (nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("CORDON", vm, "Kubernetes node did not become unschedulable");
+ }
+
+ @Override
+ public void onlineGuestResources(NodeAccess access) {
+ executeNode(access, ONLINE_HOTPLUGGED_RESOURCES);
+ }
+
+ @Override
+ public void verifyGuestResources(UserVm vm, ServiceOffering target, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ if (guestMatchesTarget(guestResources(access), target)) return;
+ sleep();
+ }
+ throw failure("GUEST_VERIFY", vm, "guest CPU or memory did not reach the target offering");
+ }
+
+ @Override
+ public boolean isKubernetesResourcesCurrent(NodeCapacitySnapshot snapshot, ServiceOffering target) {
+ GuestResources guest = new GuestResources(snapshot.getGuestOnlineCpuCount(), snapshot.getGuestMemoryKiB());
+ return guestMatchesTarget(guest, target) && kubernetesMatchesGuest(snapshot, snapshot);
+ }
+
+ @Override
+ public void restartKubelet(UserVm vm, NodeAccess access, long deadline) {
+ executeNode(access, "sudo systemctl restart kubelet");
+ while (System.currentTimeMillis() < deadline) {
+ if ("active".equals(executeNode(access, "sudo systemctl is-active kubelet").trim())) return;
+ sleep();
+ }
+ throw failure("KUBELET_RESTART", vm, "kubelet did not become active");
+ }
+
+ @Override
+ public NodeCapacitySnapshot waitForKubernetesResources(KubernetesCluster cluster, UserVm vm, ServiceOffering target,
+ NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ GuestResources guest = guestResources(access);
+ NodeCapacitySnapshot observed = nodeSnapshot(cluster, vm, access, guest);
+ if (guestMatchesTarget(guest, target) && kubernetesMatchesGuest(observed, before)) return observed;
+ sleep();
+ }
+ throw failure("CAPACITY_VERIFY", vm, "Kubernetes capacity did not match the resized guest");
+ }
+
+ @Override
+ public void restoreSchedulability(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl uncordon " + node);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "-");
+ while (System.currentTimeMillis() < deadline) {
+ if (!nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("RESTORE_SCHEDULABILITY", vm, "Kubernetes node remained unschedulable");
+ }
+
+ public static boolean capacityChanged(ServiceOffering oldOffering, ServiceOffering targetOffering) {
+ return oldOffering != null && targetOffering != null
+ && (!Objects.equals(oldOffering.getCpu(), targetOffering.getCpu())
+ || !Objects.equals(oldOffering.getRamSize(), targetOffering.getRamSize()));
+ }
+
+ static long parseCpuMillis(String value) { return value.endsWith("m") ? Long.parseLong(value.substring(0, value.length() - 1)) : Long.parseLong(value) * 1000L; }
+ static long parseMemoryBytes(String value) {
+ Matcher matcher = MEMORY_QUANTITY.matcher(value);
+ if (!matcher.matches()) throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ long number = Long.parseLong(matcher.group(1)); String unit = matcher.group(2);
+ if (unit == null) return number;
+ switch (unit) {
+ case "Ki": return number * 1024L; case "Mi": return number * 1024L * 1024L; case "Gi": return number * 1024L * 1024L * 1024L;
+ case "Ti": return number * 1024L * 1024L * 1024L * 1024L; case "K": return number * 1000L; case "M": return number * 1000L * 1000L;
+ case "G": return number * 1000L * 1000L * 1000L; case "T": return number * 1000L * 1000L * 1000L * 1000L;
+ default: throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ }
+ }
+
+ private NodeCapacitySnapshot nodeSnapshot(KubernetesCluster cluster, UserVm vm, NodeAccess access, GuestResources guest) {
+ JsonObject node = new JsonParser().parse(executeControl(access, "sudo /opt/bin/kubectl get node " + nodeName(vm) + " -o json")).getAsJsonObject();
+ JsonObject annotations = node.getAsJsonObject("metadata").has("annotations") ? node.getAsJsonObject("metadata").getAsJsonObject("annotations") : null;
+ boolean cksCordon = annotations != null && annotations.has(RESIZE_ANNOTATION) && cluster.getUuid().equals(annotations.get(RESIZE_ANNOTATION).getAsString());
+ JsonObject spec = node.has("spec") ? node.getAsJsonObject("spec") : new JsonObject(); JsonObject status = node.getAsJsonObject("status");
+ JsonObject capacity = status.getAsJsonObject("capacity"); JsonObject allocatable = status.getAsJsonObject("allocatable");
+ return new NodeCapacitySnapshot(spec.has("unschedulable") && spec.get("unschedulable").getAsBoolean(), cksCordon,
+ guest == null ? 0 : guest.cpu, guest == null ? 0 : guest.memoryKiB, parseCpuMillis(capacity.get("cpu").getAsString()),
+ parseMemoryBytes(capacity.get("memory").getAsString()), parseCpuMillis(allocatable.get("cpu").getAsString()),
+ parseMemoryBytes(allocatable.get("memory").getAsString()), isReady(status));
+ }
+
+ private boolean isReady(JsonObject status) {
+ JsonArray conditions = status.getAsJsonArray("conditions");
+ for (JsonElement condition : conditions) { JsonObject item = condition.getAsJsonObject(); if ("Ready".equals(item.get("type").getAsString()) && "True".equals(item.get("status").getAsString())) return true; }
Review Comment:
Done.
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/utils/KubernetesClusterNodeCapacityReconcilerImpl.java:
##########
@@ -0,0 +1,232 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.kubernetes.cluster.utils;
+
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import com.cloud.kubernetes.cluster.KubernetesCluster;
+import com.cloud.kubernetes.cluster.KubernetesServiceHelper.KubernetesClusterNodeType;
+import com.cloud.offering.ServiceOffering;
+import com.cloud.uservm.UserVm;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.ssh.SshHelper;
+import com.google.gson.JsonArray;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParser;
+import org.apache.commons.lang3.StringUtils;
+
+/** SSH-backed implementation for a rolling CKS live-resize reconciliation. */
+public class KubernetesClusterNodeCapacityReconcilerImpl implements KubernetesClusterNodeCapacityReconciler {
+ // KVM QA observes 4,568,788 KiB in a 4,608 MiB guest after memory hotplug
+ // (about 146 MiB below the offering), so keep a small margin above that
+ // platform overhead while still rejecting materially undersized guests.
+ static final long MINIMUM_MEMORY_OVERHEAD_BYTES = 160L * 1024L * 1024L;
+ static final long KUBERNETES_MEMORY_REPORTING_TOLERANCE_BYTES = 16L * 1024L * 1024L;
+ private static final int COMMAND_TIMEOUT_MS = 30000;
+ private static final int POLL_INTERVAL_MS = 5000;
+ private static final Pattern NODE_NAME = Pattern.compile("[A-Za-z0-9][A-Za-z0-9.-]{0,252}");
+ private static final Pattern MEMORY_QUANTITY = Pattern.compile("([0-9]+)(Ki|Mi|Gi|Ti|K|M|G|T)?");
+ private static final String RESIZE_ANNOTATION = "cloudstack.apache.org/cks-live-resize";
+ static final String ONLINE_HOTPLUGGED_RESOURCES = "sudo sh -c 'for cpu in /sys/devices/system/cpu/cpu[0-9]*; do "
+ + "[ -e \"$cpu/online\" ] || continue; echo 1 > \"$cpu/online\"; done; "
+ + "for memory in /sys/devices/system/memory/memory[0-9]*; do "
+ + "[ -e \"$memory/online\" ] || continue; echo online > \"$memory/online\"; done'";
+
+ @Override
+ public boolean requiresKubeletRefresh(ServiceOffering oldOffering, ServiceOffering newOffering,
+ KubernetesClusterNodeType nodeType, KubernetesCluster.State originalState) {
+ return KubernetesCluster.State.Running == originalState
+ && (KubernetesClusterNodeType.WORKER == nodeType || KubernetesClusterNodeType.CONTROL == nodeType)
+ && capacityChanged(oldOffering, newOffering);
+ }
+
+ @Override
+ public NodeCapacitySnapshot captureBefore(KubernetesCluster cluster, UserVm vm, NodeAccess access) {
+ return nodeSnapshot(cluster, vm, access, guestResources(access));
+ }
+
+ @Override
+ public void cordonIfNeeded(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "=" + cluster.getUuid() + " --overwrite");
+ executeControl(access, "sudo /opt/bin/kubectl cordon " + node);
+ while (System.currentTimeMillis() < deadline) {
+ if (nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("CORDON", vm, "Kubernetes node did not become unschedulable");
+ }
+
+ @Override
+ public void onlineGuestResources(NodeAccess access) {
+ executeNode(access, ONLINE_HOTPLUGGED_RESOURCES);
+ }
+
+ @Override
+ public void verifyGuestResources(UserVm vm, ServiceOffering target, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ if (guestMatchesTarget(guestResources(access), target)) return;
+ sleep();
+ }
+ throw failure("GUEST_VERIFY", vm, "guest CPU or memory did not reach the target offering");
+ }
+
+ @Override
+ public boolean isKubernetesResourcesCurrent(NodeCapacitySnapshot snapshot, ServiceOffering target) {
+ GuestResources guest = new GuestResources(snapshot.getGuestOnlineCpuCount(), snapshot.getGuestMemoryKiB());
+ return guestMatchesTarget(guest, target) && kubernetesMatchesGuest(snapshot, snapshot);
+ }
+
+ @Override
+ public void restartKubelet(UserVm vm, NodeAccess access, long deadline) {
+ executeNode(access, "sudo systemctl restart kubelet");
+ while (System.currentTimeMillis() < deadline) {
+ if ("active".equals(executeNode(access, "sudo systemctl is-active kubelet").trim())) return;
+ sleep();
+ }
+ throw failure("KUBELET_RESTART", vm, "kubelet did not become active");
+ }
+
+ @Override
+ public NodeCapacitySnapshot waitForKubernetesResources(KubernetesCluster cluster, UserVm vm, ServiceOffering target,
+ NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ GuestResources guest = guestResources(access);
+ NodeCapacitySnapshot observed = nodeSnapshot(cluster, vm, access, guest);
+ if (guestMatchesTarget(guest, target) && kubernetesMatchesGuest(observed, before)) return observed;
+ sleep();
+ }
+ throw failure("CAPACITY_VERIFY", vm, "Kubernetes capacity did not match the resized guest");
+ }
+
+ @Override
+ public void restoreSchedulability(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl uncordon " + node);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "-");
+ while (System.currentTimeMillis() < deadline) {
+ if (!nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("RESTORE_SCHEDULABILITY", vm, "Kubernetes node remained unschedulable");
+ }
+
+ public static boolean capacityChanged(ServiceOffering oldOffering, ServiceOffering targetOffering) {
+ return oldOffering != null && targetOffering != null
+ && (!Objects.equals(oldOffering.getCpu(), targetOffering.getCpu())
+ || !Objects.equals(oldOffering.getRamSize(), targetOffering.getRamSize()));
+ }
+
+ static long parseCpuMillis(String value) { return value.endsWith("m") ? Long.parseLong(value.substring(0, value.length() - 1)) : Long.parseLong(value) * 1000L; }
+ static long parseMemoryBytes(String value) {
+ Matcher matcher = MEMORY_QUANTITY.matcher(value);
+ if (!matcher.matches()) throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ long number = Long.parseLong(matcher.group(1)); String unit = matcher.group(2);
+ if (unit == null) return number;
+ switch (unit) {
+ case "Ki": return number * 1024L; case "Mi": return number * 1024L * 1024L; case "Gi": return number * 1024L * 1024L * 1024L;
+ case "Ti": return number * 1024L * 1024L * 1024L * 1024L; case "K": return number * 1000L; case "M": return number * 1000L * 1000L;
+ case "G": return number * 1000L * 1000L * 1000L; case "T": return number * 1000L * 1000L * 1000L * 1000L;
+ default: throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ }
+ }
+
+ private NodeCapacitySnapshot nodeSnapshot(KubernetesCluster cluster, UserVm vm, NodeAccess access, GuestResources guest) {
+ JsonObject node = new JsonParser().parse(executeControl(access, "sudo /opt/bin/kubectl get node " + nodeName(vm) + " -o json")).getAsJsonObject();
+ JsonObject annotations = node.getAsJsonObject("metadata").has("annotations") ? node.getAsJsonObject("metadata").getAsJsonObject("annotations") : null;
+ boolean cksCordon = annotations != null && annotations.has(RESIZE_ANNOTATION) && cluster.getUuid().equals(annotations.get(RESIZE_ANNOTATION).getAsString());
+ JsonObject spec = node.has("spec") ? node.getAsJsonObject("spec") : new JsonObject(); JsonObject status = node.getAsJsonObject("status");
+ JsonObject capacity = status.getAsJsonObject("capacity"); JsonObject allocatable = status.getAsJsonObject("allocatable");
+ return new NodeCapacitySnapshot(spec.has("unschedulable") && spec.get("unschedulable").getAsBoolean(), cksCordon,
+ guest == null ? 0 : guest.cpu, guest == null ? 0 : guest.memoryKiB, parseCpuMillis(capacity.get("cpu").getAsString()),
+ parseMemoryBytes(capacity.get("memory").getAsString()), parseCpuMillis(allocatable.get("cpu").getAsString()),
+ parseMemoryBytes(allocatable.get("memory").getAsString()), isReady(status));
+ }
+
+ private boolean isReady(JsonObject status) {
+ JsonArray conditions = status.getAsJsonArray("conditions");
+ for (JsonElement condition : conditions) { JsonObject item = condition.getAsJsonObject(); if ("Ready".equals(item.get("type").getAsString()) && "True".equals(item.get("status").getAsString())) return true; }
+ return false;
+ }
+
+ private GuestResources guestResources(NodeAccess access) {
+ String[] values = executeNode(access, "getconf _NPROCESSORS_ONLN; awk '/^MemTotal:/ {print $2}' /proc/meminfo").trim().split("\\s+");
+ if (values.length != 2) throw new CloudRuntimeException("Unable to read guest CPU and memory");
Review Comment:
Done, and I went through all the changed files, tests included, so it's consistent everywhere and not only where you marked it.
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/utils/KubernetesClusterNodeCapacityReconcilerImpl.java:
##########
@@ -0,0 +1,232 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.kubernetes.cluster.utils;
+
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import com.cloud.kubernetes.cluster.KubernetesCluster;
+import com.cloud.kubernetes.cluster.KubernetesServiceHelper.KubernetesClusterNodeType;
+import com.cloud.offering.ServiceOffering;
+import com.cloud.uservm.UserVm;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.ssh.SshHelper;
+import com.google.gson.JsonArray;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParser;
+import org.apache.commons.lang3.StringUtils;
+
+/** SSH-backed implementation for a rolling CKS live-resize reconciliation. */
+public class KubernetesClusterNodeCapacityReconcilerImpl implements KubernetesClusterNodeCapacityReconciler {
+ // KVM QA observes 4,568,788 KiB in a 4,608 MiB guest after memory hotplug
+ // (about 146 MiB below the offering), so keep a small margin above that
+ // platform overhead while still rejecting materially undersized guests.
+ static final long MINIMUM_MEMORY_OVERHEAD_BYTES = 160L * 1024L * 1024L;
+ static final long KUBERNETES_MEMORY_REPORTING_TOLERANCE_BYTES = 16L * 1024L * 1024L;
+ private static final int COMMAND_TIMEOUT_MS = 30000;
+ private static final int POLL_INTERVAL_MS = 5000;
+ private static final Pattern NODE_NAME = Pattern.compile("[A-Za-z0-9][A-Za-z0-9.-]{0,252}");
+ private static final Pattern MEMORY_QUANTITY = Pattern.compile("([0-9]+)(Ki|Mi|Gi|Ti|K|M|G|T)?");
+ private static final String RESIZE_ANNOTATION = "cloudstack.apache.org/cks-live-resize";
+ static final String ONLINE_HOTPLUGGED_RESOURCES = "sudo sh -c 'for cpu in /sys/devices/system/cpu/cpu[0-9]*; do "
+ + "[ -e \"$cpu/online\" ] || continue; echo 1 > \"$cpu/online\"; done; "
+ + "for memory in /sys/devices/system/memory/memory[0-9]*; do "
+ + "[ -e \"$memory/online\" ] || continue; echo online > \"$memory/online\"; done'";
+
+ @Override
+ public boolean requiresKubeletRefresh(ServiceOffering oldOffering, ServiceOffering newOffering,
+ KubernetesClusterNodeType nodeType, KubernetesCluster.State originalState) {
+ return KubernetesCluster.State.Running == originalState
+ && (KubernetesClusterNodeType.WORKER == nodeType || KubernetesClusterNodeType.CONTROL == nodeType)
+ && capacityChanged(oldOffering, newOffering);
+ }
+
+ @Override
+ public NodeCapacitySnapshot captureBefore(KubernetesCluster cluster, UserVm vm, NodeAccess access) {
+ return nodeSnapshot(cluster, vm, access, guestResources(access));
+ }
+
+ @Override
+ public void cordonIfNeeded(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "=" + cluster.getUuid() + " --overwrite");
+ executeControl(access, "sudo /opt/bin/kubectl cordon " + node);
+ while (System.currentTimeMillis() < deadline) {
+ if (nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("CORDON", vm, "Kubernetes node did not become unschedulable");
+ }
+
+ @Override
+ public void onlineGuestResources(NodeAccess access) {
+ executeNode(access, ONLINE_HOTPLUGGED_RESOURCES);
+ }
+
+ @Override
+ public void verifyGuestResources(UserVm vm, ServiceOffering target, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ if (guestMatchesTarget(guestResources(access), target)) return;
+ sleep();
+ }
+ throw failure("GUEST_VERIFY", vm, "guest CPU or memory did not reach the target offering");
+ }
+
+ @Override
+ public boolean isKubernetesResourcesCurrent(NodeCapacitySnapshot snapshot, ServiceOffering target) {
+ GuestResources guest = new GuestResources(snapshot.getGuestOnlineCpuCount(), snapshot.getGuestMemoryKiB());
+ return guestMatchesTarget(guest, target) && kubernetesMatchesGuest(snapshot, snapshot);
+ }
+
+ @Override
+ public void restartKubelet(UserVm vm, NodeAccess access, long deadline) {
+ executeNode(access, "sudo systemctl restart kubelet");
+ while (System.currentTimeMillis() < deadline) {
+ if ("active".equals(executeNode(access, "sudo systemctl is-active kubelet").trim())) return;
+ sleep();
+ }
+ throw failure("KUBELET_RESTART", vm, "kubelet did not become active");
+ }
+
+ @Override
+ public NodeCapacitySnapshot waitForKubernetesResources(KubernetesCluster cluster, UserVm vm, ServiceOffering target,
+ NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ GuestResources guest = guestResources(access);
+ NodeCapacitySnapshot observed = nodeSnapshot(cluster, vm, access, guest);
+ if (guestMatchesTarget(guest, target) && kubernetesMatchesGuest(observed, before)) return observed;
+ sleep();
+ }
+ throw failure("CAPACITY_VERIFY", vm, "Kubernetes capacity did not match the resized guest");
+ }
+
+ @Override
+ public void restoreSchedulability(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl uncordon " + node);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "-");
+ while (System.currentTimeMillis() < deadline) {
+ if (!nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("RESTORE_SCHEDULABILITY", vm, "Kubernetes node remained unschedulable");
+ }
+
+ public static boolean capacityChanged(ServiceOffering oldOffering, ServiceOffering targetOffering) {
+ return oldOffering != null && targetOffering != null
+ && (!Objects.equals(oldOffering.getCpu(), targetOffering.getCpu())
+ || !Objects.equals(oldOffering.getRamSize(), targetOffering.getRamSize()));
+ }
+
+ static long parseCpuMillis(String value) { return value.endsWith("m") ? Long.parseLong(value.substring(0, value.length() - 1)) : Long.parseLong(value) * 1000L; }
+ static long parseMemoryBytes(String value) {
+ Matcher matcher = MEMORY_QUANTITY.matcher(value);
+ if (!matcher.matches()) throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ long number = Long.parseLong(matcher.group(1)); String unit = matcher.group(2);
+ if (unit == null) return number;
+ switch (unit) {
+ case "Ki": return number * 1024L; case "Mi": return number * 1024L * 1024L; case "Gi": return number * 1024L * 1024L * 1024L;
+ case "Ti": return number * 1024L * 1024L * 1024L * 1024L; case "K": return number * 1000L; case "M": return number * 1000L * 1000L;
+ case "G": return number * 1000L * 1000L * 1000L; case "T": return number * 1000L * 1000L * 1000L * 1000L;
+ default: throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ }
+ }
+
+ private NodeCapacitySnapshot nodeSnapshot(KubernetesCluster cluster, UserVm vm, NodeAccess access, GuestResources guest) {
+ JsonObject node = new JsonParser().parse(executeControl(access, "sudo /opt/bin/kubectl get node " + nodeName(vm) + " -o json")).getAsJsonObject();
+ JsonObject annotations = node.getAsJsonObject("metadata").has("annotations") ? node.getAsJsonObject("metadata").getAsJsonObject("annotations") : null;
+ boolean cksCordon = annotations != null && annotations.has(RESIZE_ANNOTATION) && cluster.getUuid().equals(annotations.get(RESIZE_ANNOTATION).getAsString());
+ JsonObject spec = node.has("spec") ? node.getAsJsonObject("spec") : new JsonObject(); JsonObject status = node.getAsJsonObject("status");
+ JsonObject capacity = status.getAsJsonObject("capacity"); JsonObject allocatable = status.getAsJsonObject("allocatable");
+ return new NodeCapacitySnapshot(spec.has("unschedulable") && spec.get("unschedulable").getAsBoolean(), cksCordon,
+ guest == null ? 0 : guest.cpu, guest == null ? 0 : guest.memoryKiB, parseCpuMillis(capacity.get("cpu").getAsString()),
+ parseMemoryBytes(capacity.get("memory").getAsString()), parseCpuMillis(allocatable.get("cpu").getAsString()),
+ parseMemoryBytes(allocatable.get("memory").getAsString()), isReady(status));
+ }
+
+ private boolean isReady(JsonObject status) {
+ JsonArray conditions = status.getAsJsonArray("conditions");
+ for (JsonElement condition : conditions) { JsonObject item = condition.getAsJsonObject(); if ("Ready".equals(item.get("type").getAsString()) && "True".equals(item.get("status").getAsString())) return true; }
+ return false;
+ }
+
+ private GuestResources guestResources(NodeAccess access) {
+ String[] values = executeNode(access, "getconf _NPROCESSORS_ONLN; awk '/^MemTotal:/ {print $2}' /proc/meminfo").trim().split("\\s+");
+ if (values.length != 2) throw new CloudRuntimeException("Unable to read guest CPU and memory");
+ return new GuestResources(Long.parseLong(values[0]), Long.parseLong(values[1]));
+ }
+
+ private boolean guestMatchesTarget(GuestResources observed, ServiceOffering target) {
Review Comment:
Done, each one now has a short comment on what it compares and why.
##########
plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/utils/KubernetesClusterNodeCapacityReconcilerImpl.java:
##########
@@ -0,0 +1,232 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+package com.cloud.kubernetes.cluster.utils;
+
+import java.util.Objects;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import com.cloud.kubernetes.cluster.KubernetesCluster;
+import com.cloud.kubernetes.cluster.KubernetesServiceHelper.KubernetesClusterNodeType;
+import com.cloud.offering.ServiceOffering;
+import com.cloud.uservm.UserVm;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.ssh.SshHelper;
+import com.google.gson.JsonArray;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonParser;
+import org.apache.commons.lang3.StringUtils;
+
+/** SSH-backed implementation for a rolling CKS live-resize reconciliation. */
+public class KubernetesClusterNodeCapacityReconcilerImpl implements KubernetesClusterNodeCapacityReconciler {
+ // KVM QA observes 4,568,788 KiB in a 4,608 MiB guest after memory hotplug
+ // (about 146 MiB below the offering), so keep a small margin above that
+ // platform overhead while still rejecting materially undersized guests.
+ static final long MINIMUM_MEMORY_OVERHEAD_BYTES = 160L * 1024L * 1024L;
+ static final long KUBERNETES_MEMORY_REPORTING_TOLERANCE_BYTES = 16L * 1024L * 1024L;
+ private static final int COMMAND_TIMEOUT_MS = 30000;
+ private static final int POLL_INTERVAL_MS = 5000;
+ private static final Pattern NODE_NAME = Pattern.compile("[A-Za-z0-9][A-Za-z0-9.-]{0,252}");
+ private static final Pattern MEMORY_QUANTITY = Pattern.compile("([0-9]+)(Ki|Mi|Gi|Ti|K|M|G|T)?");
+ private static final String RESIZE_ANNOTATION = "cloudstack.apache.org/cks-live-resize";
+ static final String ONLINE_HOTPLUGGED_RESOURCES = "sudo sh -c 'for cpu in /sys/devices/system/cpu/cpu[0-9]*; do "
+ + "[ -e \"$cpu/online\" ] || continue; echo 1 > \"$cpu/online\"; done; "
+ + "for memory in /sys/devices/system/memory/memory[0-9]*; do "
+ + "[ -e \"$memory/online\" ] || continue; echo online > \"$memory/online\"; done'";
+
+ @Override
+ public boolean requiresKubeletRefresh(ServiceOffering oldOffering, ServiceOffering newOffering,
+ KubernetesClusterNodeType nodeType, KubernetesCluster.State originalState) {
+ return KubernetesCluster.State.Running == originalState
+ && (KubernetesClusterNodeType.WORKER == nodeType || KubernetesClusterNodeType.CONTROL == nodeType)
+ && capacityChanged(oldOffering, newOffering);
+ }
+
+ @Override
+ public NodeCapacitySnapshot captureBefore(KubernetesCluster cluster, UserVm vm, NodeAccess access) {
+ return nodeSnapshot(cluster, vm, access, guestResources(access));
+ }
+
+ @Override
+ public void cordonIfNeeded(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "=" + cluster.getUuid() + " --overwrite");
+ executeControl(access, "sudo /opt/bin/kubectl cordon " + node);
+ while (System.currentTimeMillis() < deadline) {
+ if (nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("CORDON", vm, "Kubernetes node did not become unschedulable");
+ }
+
+ @Override
+ public void onlineGuestResources(NodeAccess access) {
+ executeNode(access, ONLINE_HOTPLUGGED_RESOURCES);
+ }
+
+ @Override
+ public void verifyGuestResources(UserVm vm, ServiceOffering target, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ if (guestMatchesTarget(guestResources(access), target)) return;
+ sleep();
+ }
+ throw failure("GUEST_VERIFY", vm, "guest CPU or memory did not reach the target offering");
+ }
+
+ @Override
+ public boolean isKubernetesResourcesCurrent(NodeCapacitySnapshot snapshot, ServiceOffering target) {
+ GuestResources guest = new GuestResources(snapshot.getGuestOnlineCpuCount(), snapshot.getGuestMemoryKiB());
+ return guestMatchesTarget(guest, target) && kubernetesMatchesGuest(snapshot, snapshot);
+ }
+
+ @Override
+ public void restartKubelet(UserVm vm, NodeAccess access, long deadline) {
+ executeNode(access, "sudo systemctl restart kubelet");
+ while (System.currentTimeMillis() < deadline) {
+ if ("active".equals(executeNode(access, "sudo systemctl is-active kubelet").trim())) return;
+ sleep();
+ }
+ throw failure("KUBELET_RESTART", vm, "kubelet did not become active");
+ }
+
+ @Override
+ public NodeCapacitySnapshot waitForKubernetesResources(KubernetesCluster cluster, UserVm vm, ServiceOffering target,
+ NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ while (System.currentTimeMillis() < deadline) {
+ GuestResources guest = guestResources(access);
+ NodeCapacitySnapshot observed = nodeSnapshot(cluster, vm, access, guest);
+ if (guestMatchesTarget(guest, target) && kubernetesMatchesGuest(observed, before)) return observed;
+ sleep();
+ }
+ throw failure("CAPACITY_VERIFY", vm, "Kubernetes capacity did not match the resized guest");
+ }
+
+ @Override
+ public void restoreSchedulability(KubernetesCluster cluster, UserVm vm, NodeCapacitySnapshot before, NodeAccess access, long deadline) {
+ if (before.isUnschedulable() && !before.isCloudStackResizeCordon()) return;
+ String node = nodeName(vm);
+ executeControl(access, "sudo /opt/bin/kubectl uncordon " + node);
+ executeControl(access, "sudo /opt/bin/kubectl annotate node " + node + " " + RESIZE_ANNOTATION + "-");
+ while (System.currentTimeMillis() < deadline) {
+ if (!nodeSnapshot(cluster, vm, access, null).isUnschedulable()) return;
+ sleep();
+ }
+ throw failure("RESTORE_SCHEDULABILITY", vm, "Kubernetes node remained unschedulable");
+ }
+
+ public static boolean capacityChanged(ServiceOffering oldOffering, ServiceOffering targetOffering) {
+ return oldOffering != null && targetOffering != null
+ && (!Objects.equals(oldOffering.getCpu(), targetOffering.getCpu())
+ || !Objects.equals(oldOffering.getRamSize(), targetOffering.getRamSize()));
+ }
+
+ static long parseCpuMillis(String value) { return value.endsWith("m") ? Long.parseLong(value.substring(0, value.length() - 1)) : Long.parseLong(value) * 1000L; }
+ static long parseMemoryBytes(String value) {
+ Matcher matcher = MEMORY_QUANTITY.matcher(value);
+ if (!matcher.matches()) throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ long number = Long.parseLong(matcher.group(1)); String unit = matcher.group(2);
+ if (unit == null) return number;
+ switch (unit) {
+ case "Ki": return number * 1024L; case "Mi": return number * 1024L * 1024L; case "Gi": return number * 1024L * 1024L * 1024L;
+ case "Ti": return number * 1024L * 1024L * 1024L * 1024L; case "K": return number * 1000L; case "M": return number * 1000L * 1000L;
+ case "G": return number * 1000L * 1000L * 1000L; case "T": return number * 1000L * 1000L * 1000L * 1000L;
+ default: throw new IllegalArgumentException("Unsupported Kubernetes memory quantity");
+ }
+ }
+
+ private NodeCapacitySnapshot nodeSnapshot(KubernetesCluster cluster, UserVm vm, NodeAccess access, GuestResources guest) {
+ JsonObject node = new JsonParser().parse(executeControl(access, "sudo /opt/bin/kubectl get node " + nodeName(vm) + " -o json")).getAsJsonObject();
+ JsonObject annotations = node.getAsJsonObject("metadata").has("annotations") ? node.getAsJsonObject("metadata").getAsJsonObject("annotations") : null;
+ boolean cksCordon = annotations != null && annotations.has(RESIZE_ANNOTATION) && cluster.getUuid().equals(annotations.get(RESIZE_ANNOTATION).getAsString());
+ JsonObject spec = node.has("spec") ? node.getAsJsonObject("spec") : new JsonObject(); JsonObject status = node.getAsJsonObject("status");
+ JsonObject capacity = status.getAsJsonObject("capacity"); JsonObject allocatable = status.getAsJsonObject("allocatable");
+ return new NodeCapacitySnapshot(spec.has("unschedulable") && spec.get("unschedulable").getAsBoolean(), cksCordon,
+ guest == null ? 0 : guest.cpu, guest == null ? 0 : guest.memoryKiB, parseCpuMillis(capacity.get("cpu").getAsString()),
+ parseMemoryBytes(capacity.get("memory").getAsString()), parseCpuMillis(allocatable.get("cpu").getAsString()),
+ parseMemoryBytes(allocatable.get("memory").getAsString()), isReady(status));
+ }
+
+ private boolean isReady(JsonObject status) {
+ JsonArray conditions = status.getAsJsonArray("conditions");
+ for (JsonElement condition : conditions) { JsonObject item = condition.getAsJsonObject(); if ("Ready".equals(item.get("type").getAsString()) && "True".equals(item.get("status").getAsString())) return true; }
+ return false;
+ }
+
+ private GuestResources guestResources(NodeAccess access) {
+ String[] values = executeNode(access, "getconf _NPROCESSORS_ONLN; awk '/^MemTotal:/ {print $2}' /proc/meminfo").trim().split("\\s+");
+ if (values.length != 2) throw new CloudRuntimeException("Unable to read guest CPU and memory");
+ return new GuestResources(Long.parseLong(values[0]), Long.parseLong(values[1]));
+ }
+
+ private boolean guestMatchesTarget(GuestResources observed, ServiceOffering target) {
+ long targetMemory = target.getRamSize() * 1024L * 1024L; long observedMemory = observed.memoryKiB * 1024L;
+ long overhead = Math.max(targetMemory / 50L, MINIMUM_MEMORY_OVERHEAD_BYTES);
+ return observed.cpu == target.getCpu() && observedMemory <= targetMemory && targetMemory - observedMemory <= overhead;
+ }
+
+ private boolean kubernetesMatchesGuest(NodeCapacitySnapshot observed, NodeCapacitySnapshot before) {
+ return observed.isReady() && observed.getCapacityCpuMillis() == observed.getGuestOnlineCpuCount() * 1000L
+ && Math.abs(observed.getCapacityMemoryBytes() - observed.getGuestMemoryKiB() * 1024L) <= KUBERNETES_MEMORY_REPORTING_TOLERANCE_BYTES
+ && (observed.getGuestOnlineCpuCount() <= before.getGuestOnlineCpuCount() || observed.getAllocatableCpuMillis() > before.getAllocatableCpuMillis())
+ && (observed.getGuestMemoryKiB() <= before.getGuestMemoryKiB() || observed.getAllocatableMemoryBytes() > before.getAllocatableMemoryBytes());
+ }
+
+ protected String executeControl(NodeAccess access, String command) {
+ return execute(access.getControlAddress(), access.getControlPort(), access, command);
+ }
+
+ protected String executeNode(NodeAccess access, String command) {
+ return execute(access.getNodeAddress(), access.getNodePort(), access, command);
+ }
+
+ private String execute(String address, int port, NodeAccess access, String command) {
+ Pair<Boolean, String> result;
+ try {
+ result = SshHelper.sshExecute(address, port, access.getUser(), access.getSshKeyFile(), null, command, 10000, 10000, COMMAND_TIMEOUT_MS);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new NodeCapacityReconciliationException("CKS live-resize command interrupted", e);
+ } catch (RuntimeException e) {
+ throw e;
+ } catch (Exception e) {
+ // SshHelper declares Exception, including authentication failures.
+ // Translate only at this legacy boundary, not around the resize workflow.
+ throw new NodeCapacityReconciliationException("CKS live-resize SSH command failed", e);
+ }
+ if (!Boolean.TRUE.equals(result.first())) {
+ throw new NodeCapacityReconciliationException("CKS live-resize command failed");
+ }
+ return StringUtils.defaultString(result.second());
+ }
+ private String nodeName(UserVm vm) { String name = StringUtils.lowerCase(vm.getHostName()); if (!NODE_NAME.matcher(name).matches()) throw new CloudRuntimeException("Invalid Kubernetes node name for live resize"); return name; }
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6455959471
## [](https://sonarcloud.io/dashboard?id=apache_cloudstack&pullRequest=14223) **Quality Gate passed**
Issues
 [9 New issues](https://sonarcloud.io/project/issues?id=apache_cloudstack&pullRequest=14223&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_cloudstack&pullRequest=14223&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_cloudstack&pullRequest=14223&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [72.7% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_cloudstack&pullRequest=14223&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_cloudstack&pullRequest=14223&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_cloudstack&pullRequest=14223)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "andrijapanicsb (via GitHub) via commits" <co...@cloudstack.apache.org>.
andrijapanicsb commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6704851201
Retested the published packages from `c8e9058956`: **35 of 36 functional and regression tests passed**. The only exception was the known issue #14552: autoscaling without an explicit offering still falls back to the old cluster-wide offering.
Both workers brought the delayed CPU online automatically. The UI retry preserved the operator cordon, and the final CPU and RAM totals matched the API and Kubernetes checks.
The attached PDF contains the full test matrix and screenshots.
[cloudstack-pr-14223-functional-testing-c8e9058956-20261107.pdf](https://github.com/user-attachments/files/34514656/cloudstack-pr-14223-functional-testing-c8e9058956-20261107.pdf)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "andrijapanicsb (via GitHub) via commits" <co...@cloudstack.apache.org>.
andrijapanicsb commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6706137927
@blueorangutan package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "ACSHomeBot (via GitHub) via commits" <co...@cloudstack.apache.org>.
ACSHomeBot commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6703050829
### Packaging results:
| Result | Artifact | Platform | Build Time |
|:--|:--|:--|--:|
| PASS | RPM | EL (EL8/9/10) | 29 min |
| PASS | DEB | Ubuntu, Debian | 21 min |
**Test packages are available at:**
- EL9/RPM: [https://packages.v2kmigrate.com/cloudstack/pr/14223/el9/](https://packages.v2kmigrate.com/cloudstack/pr/14223/el9/index.html)
- Debian/APT: [https://packages.v2kmigrate.com/cloudstack/pr/14223/debian/](https://packages.v2kmigrate.com/cloudstack/pr/14223/debian/index.html)
<!-- acshomebot-package-results:20261107T045940Z-ec53d659:1 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CKS: reconcile node capacity after live resize [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on PR #14223:
URL: https://github.com/apache/cloudstack/pull/14223#issuecomment-6504422486
@blueorangutan test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org