You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by GitBox <gi...@apache.org> on 2022/02/10 06:51:46 UTC
[GitHub] [cloudstack] satishglondhe opened a new issue #5976: Kubernetes Auto Scaling not working in version 4.16.0.0
satishglondhe opened a new issue #5976:
URL: https://github.com/apache/cloudstack/issues/5976
<!--
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release and main branch are affected too.
Always add information AFTER of these HTML comments, but no need to delete the comments.
-->
##### ISSUE TYPE
<!-- Pick one below and delete the rest -->
* Bug Report
* Other
##### COMPONENT NAME
<!--
Categorize the issue, e.g. API, VR, VPN, UI, etc.
-->
~~~
UI
~~~
##### CLOUDSTACK VERSION
<!--
New line separated list of affected versions, commit ID for issues on main branch.
-->
~~~
4.16.0.0
~~~
##### CONFIGURATION
<!--
Information about the configuration if relevant, e.g. basic network, advanced networking, etc. N/A otherwise
-->
Advanced Network
##### OS / ENVIRONMENT
<!--
Information about the environment if relevant, N/A otherwise
-->
N/A
##### SUMMARY
<!-- Explain the problem/feature briefly -->
After successful creation of kubernetes cluster, and selecting autoscaling, specifying min and max size of worker nodes, Autoscaling pod are generated in kube-system, but it doesn't function.
While checking for logs of cluster-autoscaler pod in kube-system we get the following logs.
I0131 17:01:11.537869 1 client.go:175] NewAPIRequest response status code:200
I0131 17:01:11.538288 1 cloudstack_manager.go:88] Got cluster : &{c6496415-daea-4051-993a-3b9f9c58242e voyager 1 3 2
1 [0xc000e889f0 0xc000e88a20 0xc000e88a50]}
E0131 17:01:11.538366 1 static_autoscaler.go:271] Failed to get node infos for groups: Unable to find node voyager-c
ontrol-17ea9f68878 in cluster
I0131 17:01:21.538749 1 static_autoscaler.go:229] Starting main loop
It seems like, it failed to get node infos for groups.
##### STEPS TO REPRODUCE
<!--
For bugs, show exactly how to reproduce the problem, using a minimal test-case. Use Screenshots if accurate.
For new features, show how the feature would be used.
-->
<!-- Paste example playbooks or commands between quotes below -->
~~~
Create a kubernetes cluster with versions [1.22.2, 1.20.9, 1.21.3]
Enable auto scaling and specify min and max cluster size
~~~
<!-- You can also paste gist.github.com links for larger files -->
##### EXPECTED RESULTS
<!-- What did you expect to happen when running the steps above? -->
~~~
After increased load in cluster, cluster autoscaler should increase the node and assign the pending pods on this new node. Same, for scale down.
~~~
##### ACTUAL RESULTS
<!-- What actually happened? -->
<!-- Paste verbatim command output between quotes below -->
~~~
After increased load in cluster, cluster autoscaler is not provisioning new nodes
~~~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] davidjumani commented on issue #5976: Kubernetes Auto Scaling not working in version 4.16.0.0
Posted by GitBox <gi...@apache.org>.
davidjumani commented on issue #5976:
URL: https://github.com/apache/cloudstack/issues/5976#issuecomment-1059017125
@satishglondhe Can you try creating a cluster from the latest v1.22 / 1.23 binaries here
http://download.cloudstack.org/cks/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] davidjumani commented on issue #5976: Kubernetes Auto Scaling not working in version 4.16.0.0
Posted by GitBox <gi...@apache.org>.
davidjumani commented on issue #5976:
URL: https://github.com/apache/cloudstack/issues/5976#issuecomment-1050567927
@satishglondhe Does the kubernetes node names match the VM names ? Also can you post a full log cycle. Sometimes it might not detect the node via the uuid but does so by the hostname. You can also try creating an HPA and increasing the load and watch the autoscaler to see if it detects that pods are unschedulable and comes up with a plan to scale up the cluster similar to :
```
I1023 10:24:21.347588 1 klogx.go:86] Pod default/php-apache-79544c9bd9-fgvlg is unschedulable
I1023 10:24:21.347591 1 klogx.go:86] Pod default/php-apache-79544c9bd9-5rphq is unschedulable
I1023 10:24:21.347593 1 klogx.go:86] Pod default/php-apache-79544c9bd9-bc272 is unschedulable
I1023 10:24:21.347596 1 klogx.go:86] Pod default/php-apache-79544c9bd9-44svc is unschedulable
I1023 10:24:21.347598 1 klogx.go:86] Pod default/php-apache-79544c9bd9-nlzhr is unschedulable
I1023 10:24:21.347601 1 klogx.go:86] Pod default/php-apache-79544c9bd9-fctw6 is unschedulable
I1023 10:24:21.347616 1 scale_up.go:364] Upcoming 0 nodes
I1023 10:24:21.348158 1 scale_up.go:456] Best option to resize: 43625790-9620-45c1-95e0-c6316d0117e0
I1023 10:24:21.348166 1 scale_up.go:460] Estimated 2 nodes needed in 43625790-9620-45c1-95e0-c6316d0117e0
I1023 10:24:21.348176 1 scale_up.go:574] Final scale-up plan: [{43625790-9620-45c1-95e0-c6316d0117e0 1->3 (max: 3)}]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] davidjumani edited a comment on issue #5976: Kubernetes Auto Scaling not working in version 4.16.0.0
Posted by GitBox <gi...@apache.org>.
davidjumani edited a comment on issue #5976:
URL: https://github.com/apache/cloudstack/issues/5976#issuecomment-1038973161
@satishglondhe Can you describe the autoscaler deployment? And also set the imagePullPolicy to Always and then destroy the pod ?
Can you also list the VMs in the cluster on cloudstack and k8s and check whether the names match ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] satishglondhe commented on issue #5976: Kubernetes Auto Scaling not working in version 4.16.0.0
Posted by GitBox <gi...@apache.org>.
satishglondhe commented on issue #5976:
URL: https://github.com/apache/cloudstack/issues/5976#issuecomment-1060236491
@davidjumani We tried with the latest k8s versions 1.22, 1.23 you mentioned. No, autoscaling is still not working accordingly. Getting same errors, failed to get node infos. Does this have to do with the hypervisor. We're using xcp-ng 8.2, DBV 0.0.1, ACS version- 4.16.0.0. Also can you mention, what kind of setup you're using, or what is ideal for autoscaling to work in k8s. We can implement the same, and give it a try.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] davidjumani edited a comment on issue #5976: Kubernetes Auto Scaling not working in version 4.16.0.0
Posted by GitBox <gi...@apache.org>.
davidjumani edited a comment on issue #5976:
URL: https://github.com/apache/cloudstack/issues/5976#issuecomment-1059017125
@satishglondhe Can you try creating a cluster from the latest v1.22 / 1.23 binaries here and retry autoscaling the cluster
http://download.cloudstack.org/cks/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] davidjumani edited a comment on issue #5976: Kubernetes Auto Scaling not working in version 4.16.0.0
Posted by GitBox <gi...@apache.org>.
davidjumani edited a comment on issue #5976:
URL: https://github.com/apache/cloudstack/issues/5976#issuecomment-1066514021
@satishglondhe Looks to be an issue between the UUID of the VM on the hypervisor vs CloudStack. I've made some changes to the autoscaler image `davidjumani/cloudstack-kubernetes-autoscaler:name-dev`. Can you destroy the container, pull and try again ?
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] davidjumani commented on issue #5976: Kubernetes Auto Scaling not working in version 4.16.0.0
Posted by GitBox <gi...@apache.org>.
davidjumani commented on issue #5976:
URL: https://github.com/apache/cloudstack/issues/5976#issuecomment-1047423202
Can you edit the deployment and point the container image to `apache/cloudstack-kubernetes-autoscaler:latest`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] davidjumani closed issue #5976: Kubernetes Auto Scaling not working in version 4.16.0.0
Posted by GitBox <gi...@apache.org>.
davidjumani closed issue #5976:
URL: https://github.com/apache/cloudstack/issues/5976
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] davidjumani edited a comment on issue #5976: Kubernetes Auto Scaling not working in version 4.16.0.0
Posted by GitBox <gi...@apache.org>.
davidjumani edited a comment on issue #5976:
URL: https://github.com/apache/cloudstack/issues/5976#issuecomment-1066514021
@satishglondhe Looks to be an issue between the UUID of the VM on the hypervisor vs CloudStack. I've made some changes to the autoscaler image. Can you destroy the container, pull and try again ?
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] davidjumani edited a comment on issue #5976: Kubernetes Auto Scaling not working in version 4.16.0.0
Posted by GitBox <gi...@apache.org>.
davidjumani edited a comment on issue #5976:
URL: https://github.com/apache/cloudstack/issues/5976#issuecomment-1050567927
@satishglondhe Does the kubernetes node names match the VM names ? Also can you post a full log cycle. Sometimes it might not detect the node via the uuid but does so by the hostname. You can also try creating an HPA and increasing the load and watch the autoscaler to see if it detects that pods are unschedulable and comes up with a plan to scale up the cluster similar to :
```
I1023 10:24:21.347588 1 klogx.go:86] Pod default/php-apache-79544c9bd9-fgvlg is unschedulable
I1023 10:24:21.347591 1 klogx.go:86] Pod default/php-apache-79544c9bd9-5rphq is unschedulable
I1023 10:24:21.347593 1 klogx.go:86] Pod default/php-apache-79544c9bd9-bc272 is unschedulable
I1023 10:24:21.347596 1 klogx.go:86] Pod default/php-apache-79544c9bd9-44svc is unschedulable
I1023 10:24:21.347598 1 klogx.go:86] Pod default/php-apache-79544c9bd9-nlzhr is unschedulable
I1023 10:24:21.347601 1 klogx.go:86] Pod default/php-apache-79544c9bd9-fctw6 is unschedulable
I1023 10:24:21.347616 1 scale_up.go:364] Upcoming 0 nodes
I1023 10:24:21.348158 1 scale_up.go:456] Best option to resize: 43625790-9620-45c1-95e0-c6316d0117e0
I1023 10:24:21.348166 1 scale_up.go:460] Estimated 2 nodes needed in 43625790-9620-45c1-95e0-c6316d0117e0
I1023 10:24:21.348176 1 scale_up.go:574] Final scale-up plan: [{43625790-9620-45c1-95e0-c6316d0117e0 1->3 (max: 3)}]
```
Edit: You'd need to deploy the metrics server for HPA to work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] satishglondhe commented on issue #5976: Kubernetes Auto Scaling not working in version 4.16.0.0
Posted by GitBox <gi...@apache.org>.
satishglondhe commented on issue #5976:
URL: https://github.com/apache/cloudstack/issues/5976#issuecomment-1053948242
Yes, the Kubernetes nodes' names and the names of VMs in the cloudstack UI match. I tried creating HPA, the metrics server as you mentioned, increased the load, following this article https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/ . I was able to see, hpa creating new pods when load was increased. But, after the resources on the node reached it's limit, the remaining pods went into pending state. I monitored the cluster autoscaler logs for quite a while, but the logs only mentioned failed to get node info groups. It did not detect that the pods were unschedulable, and didn't come up with a plan to scale. I am attaching the log cycle for cluster autoscaler pod.
root@ip-172-31-92-239:~# kubectl logs -f cluster-autoscaler-6fdd97ff58-c5nw2 -n kube-system
I0226 19:37:39.546044 1 flags.go:52] FLAG: --add-dir-header="false"
I0226 19:37:39.546130 1 flags.go:52] FLAG: --address=":8085"
I0226 19:37:39.546149 1 flags.go:52] FLAG: --alsologtostderr="false"
I0226 19:37:39.546163 1 flags.go:52] FLAG: --aws-use-static-instance-list="false"
I0226 19:37:39.546175 1 flags.go:52] FLAG: --balance-similar-node-groups="false"
I0226 19:37:39.546188 1 flags.go:52] FLAG: --balancing-ignore-label="[]"
I0226 19:37:39.546200 1 flags.go:52] FLAG: --cloud-config="/config/cloud-config"
I0226 19:37:39.546213 1 flags.go:52] FLAG: --cloud-provider="cloudstack"
I0226 19:37:39.546250 1 flags.go:52] FLAG: --cloud-provider-gce-l7lb-src-cidrs="130.211.0.0/22,35.191.0.0/16"
I0226 19:37:39.546277 1 flags.go:52] FLAG: --cloud-provider-gce-lb-src-cidrs="130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16"
I0226 19:37:39.546302 1 flags.go:52] FLAG: --cluster-name=""
I0226 19:37:39.546315 1 flags.go:52] FLAG: --clusterapi-cloud-config-authoritative="false"
I0226 19:37:39.546328 1 flags.go:52] FLAG: --cores-total="0:320000"
I0226 19:37:39.546348 1 flags.go:52] FLAG: --estimator="binpacking"
I0226 19:37:39.546386 1 flags.go:52] FLAG: --expander="random"
I0226 19:37:39.546417 1 flags.go:52] FLAG: --expendable-pods-priority-cutoff="-10"
I0226 19:37:39.546439 1 flags.go:52] FLAG: --gpu-total="[]"
I0226 19:37:39.546453 1 flags.go:52] FLAG: --ignore-daemonsets-utilization="false"
I0226 19:37:39.546473 1 flags.go:52] FLAG: --ignore-mirror-pods-utilization="false"
I0226 19:37:39.546486 1 flags.go:52] FLAG: --ignore-taint="[]"
I0226 19:37:39.546499 1 flags.go:52] FLAG: --kubeconfig=""
I0226 19:37:39.546534 1 flags.go:52] FLAG: --kubernetes=""
I0226 19:37:39.546564 1 flags.go:52] FLAG: --leader-elect="true"
I0226 19:37:39.546587 1 flags.go:52] FLAG: --leader-elect-lease-duration="15s"
I0226 19:37:39.546601 1 flags.go:52] FLAG: --leader-elect-renew-deadline="10s"
I0226 19:37:39.546614 1 flags.go:52] FLAG: --leader-elect-resource-lock="leases"
I0226 19:37:39.546628 1 flags.go:52] FLAG: --leader-elect-resource-name=""
I0226 19:37:39.546648 1 flags.go:52] FLAG: --leader-elect-resource-namespace=""
I0226 19:37:39.546679 1 flags.go:52] FLAG: --leader-elect-retry-period="2s"
I0226 19:37:39.546709 1 flags.go:52] FLAG: --log-backtrace-at=":0"
I0226 19:37:39.546727 1 flags.go:52] FLAG: --log-dir=""
I0226 19:37:39.546749 1 flags.go:52] FLAG: --log-file=""
I0226 19:37:39.546762 1 flags.go:52] FLAG: --log-file-max-size="1800"
I0226 19:37:39.546775 1 flags.go:52] FLAG: --logtostderr="true"
I0226 19:37:39.546795 1 flags.go:52] FLAG: --max-autoprovisioned-node-group-count="15"
I0226 19:37:39.546829 1 flags.go:52] FLAG: --max-bulk-soft-taint-count="10"
I0226 19:37:39.546854 1 flags.go:52] FLAG: --max-bulk-soft-taint-time="3s"
I0226 19:37:39.546876 1 flags.go:52] FLAG: --max-empty-bulk-delete="10"
I0226 19:37:39.546889 1 flags.go:52] FLAG: --max-failing-time="15m0s"
I0226 19:37:39.546902 1 flags.go:52] FLAG: --max-graceful-termination-sec="600"
I0226 19:37:39.546914 1 flags.go:52] FLAG: --max-inactivity="10m0s"
I0226 19:37:39.546934 1 flags.go:52] FLAG: --max-node-provision-time="15m0s"
I0226 19:37:39.546965 1 flags.go:52] FLAG: --max-nodes-total="0"
I0226 19:37:39.546995 1 flags.go:52] FLAG: --max-total-unready-percentage="45"
I0226 19:37:39.547017 1 flags.go:52] FLAG: --memory-total="0:6400000"
I0226 19:37:39.547032 1 flags.go:52] FLAG: --min-replica-count="0"
I0226 19:37:39.547044 1 flags.go:52] FLAG: --namespace="kube-system"
I0226 19:37:39.547057 1 flags.go:52] FLAG: --new-pod-scale-up-delay="0s"
I0226 19:37:39.547069 1 flags.go:52] FLAG: --node-autoprovisioning-enabled="false"
I0226 19:37:39.547105 1 flags.go:52] FLAG: --node-deletion-delay-timeout="2m0s"
I0226 19:37:39.547135 1 flags.go:52] FLAG: --node-group-auto-discovery="[]"
I0226 19:37:39.547157 1 flags.go:52] FLAG: --nodes="[1:3:1038d730-7444-4fde-bf1b-9f86712943df]"
I0226 19:37:39.547171 1 flags.go:52] FLAG: --ok-total-unready-count="3"
I0226 19:37:39.547191 1 flags.go:52] FLAG: --profiling="false"
I0226 19:37:39.547204 1 flags.go:52] FLAG: --regional="false"
I0226 19:37:39.547216 1 flags.go:52] FLAG: --scale-down-candidates-pool-min-count="50"
I0226 19:37:39.547251 1 flags.go:52] FLAG: --scale-down-candidates-pool-ratio="0.1"
I0226 19:37:39.547281 1 flags.go:52] FLAG: --scale-down-delay-after-add="10m0s"
I0226 19:37:39.547302 1 flags.go:52] FLAG: --scale-down-delay-after-delete="0s"
I0226 19:37:39.547316 1 flags.go:52] FLAG: --scale-down-delay-after-failure="3m0s"
I0226 19:37:39.547328 1 flags.go:52] FLAG: --scale-down-enabled="true"
I0226 19:37:39.547348 1 flags.go:52] FLAG: --scale-down-gpu-utilization-threshold="0.5"
I0226 19:37:39.547362 1 flags.go:52] FLAG: --scale-down-non-empty-candidates-count="30"
I0226 19:37:39.547393 1 flags.go:52] FLAG: --scale-down-unneeded-time="10m0s"
I0226 19:37:39.547423 1 flags.go:52] FLAG: --scale-down-unready-time="20m0s"
I0226 19:37:39.547438 1 flags.go:52] FLAG: --scale-down-utilization-threshold="0.5"
I0226 19:37:39.547451 1 flags.go:52] FLAG: --scale-up-from-zero="true"
I0226 19:37:39.547463 1 flags.go:52] FLAG: --scan-interval="10s"
I0226 19:37:39.547475 1 flags.go:52] FLAG: --skip-headers="false"
I0226 19:37:39.547488 1 flags.go:52] FLAG: --skip-log-headers="false"
I0226 19:37:39.547508 1 flags.go:52] FLAG: --skip-nodes-with-local-storage="false"
I0226 19:37:39.547543 1 flags.go:52] FLAG: --skip-nodes-with-system-pods="true"
I0226 19:37:39.547567 1 flags.go:52] FLAG: --stderrthreshold="0"
I0226 19:37:39.547589 1 flags.go:52] FLAG: --unremovable-node-recheck-timeout="5m0s"
I0226 19:37:39.547618 1 flags.go:52] FLAG: --v="4"
I0226 19:37:39.547644 1 flags.go:52] FLAG: --vmodule=""
I0226 19:37:39.547692 1 flags.go:52] FLAG: --write-status-configmap="true"
I0226 19:37:39.547723 1 main.go:378] Cluster Autoscaler 1.19.0-beta.1
I0226 19:37:39.557847 1 leaderelection.go:246] attempting to acquire leader lease kube-system/cluster-autoscaler...
I0226 19:37:39.560045 1 leaderelection.go:345] lock is held by cluster-autoscaler-6f546b8c58-g74wq and has not yet expired
I0226 19:37:39.560103 1 leaderelection.go:251] failed to acquire lease kube-system/cluster-autoscaler
I0226 19:37:43.014256 1 leaderelection.go:345] lock is held by cluster-autoscaler-6f546b8c58-g74wq and has not yet expired
I0226 19:37:43.014273 1 leaderelection.go:251] failed to acquire lease kube-system/cluster-autoscaler
I0226 19:37:47.274930 1 leaderelection.go:345] lock is held by cluster-autoscaler-6f546b8c58-g74wq and has not yet expired
I0226 19:37:47.274944 1 leaderelection.go:251] failed to acquire lease kube-system/cluster-autoscaler
I0226 19:37:50.872948 1 leaderelection.go:345] lock is held by cluster-autoscaler-6f546b8c58-g74wq and has not yet expired
I0226 19:37:50.872961 1 leaderelection.go:251] failed to acquire lease kube-system/cluster-autoscaler
I0226 19:37:53.926798 1 leaderelection.go:345] lock is held by cluster-autoscaler-6f546b8c58-g74wq and has not yet expired
I0226 19:37:53.926944 1 leaderelection.go:251] failed to acquire lease kube-system/cluster-autoscaler
I0226 19:37:56.953890 1 leaderelection.go:256] successfully acquired lease kube-system/cluster-autoscaler
I0226 19:37:56.954242 1 event_sink_logging_wrapper.go:48] Event(v1.ObjectReference{Kind:"Lease", Namespace:"kube-system", Name:"cluster-autoscaler", UID:"f8780dc2-891a-4515-9a73-d6e4de715780", APIVersion:"coordination.k8s.io/v1", ResourceVersion:"64606", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' cluster-autoscaler-6fdd97ff58-c5nw2 became leader
I0226 19:37:56.958415 1 reflector.go:210] Starting reflector *v1.Pod (1h0m0s) from /home/djumani/lab/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:188
I0226 19:37:56.958755 1 reflector.go:246] Listing and watching *v1.Pod from /home/djumani/lab/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:188
I0226 19:37:56.958631 1 reflector.go:210] Starting reflector *v1.Pod (1h0m0s) from /home/djumani/lab/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:212
I0226 19:37:56.959304 1 reflector.go:246] Listing and watching *v1.Pod from /home/djumani/lab/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:212
I0226 19:37:56.958671 1 reflector.go:210] Starting reflector *v1.Node (1h0m0s) from /home/djumani/lab/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:246
I0226 19:37:56.959468 1 reflector.go:246] Listing and watching *v1.Node from /home/djumani/lab/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:246
I0226 19:37:56.958680 1 reflector.go:210] Starting reflector *v1.Node (1h0m0s) from /home/djumani/lab/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:246
I0226 19:37:56.959610 1 reflector.go:246] Listing and watching *v1.Node from /home/djumani/lab/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:246
I0226 19:37:56.958686 1 reflector.go:210] Starting reflector *v1beta1.PodDisruptionBudget (1h0m0s) from /home/djumani/lab/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:309
I0226 19:37:56.959774 1 reflector.go:246] Listing and watching *v1beta1.PodDisruptionBudget from /home/djumani/lab/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:309
I0226 19:37:56.958699 1 reflector.go:210] Starting reflector *v1.DaemonSet (1h0m0s) from /home/djumani/lab/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:320
I0226 19:37:56.959956 1 reflector.go:246] Listing and watching *v1.DaemonSet from /home/djumani/lab/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:320
I0226 19:37:56.958707 1 reflector.go:210] Starting reflector *v1.ReplicationController (1h0m0s) from /home/djumani/lab/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:329
I0226 19:37:56.960207 1 reflector.go:246] Listing and watching *v1.ReplicationController from /home/djumani/lab/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:329
I0226 19:37:56.958725 1 reflector.go:210] Starting reflector *v1.Job (1h0m0s) from /home/djumani/lab/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:338
I0226 19:37:56.960784 1 reflector.go:246] Listing and watching *v1.Job from /home/djumani/lab/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:338
I0226 19:37:56.958733 1 reflector.go:210] Starting reflector *v1.ReplicaSet (1h0m0s) from /home/djumani/lab/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:347
I0226 19:37:56.961023 1 reflector.go:246] Listing and watching *v1.ReplicaSet from /home/djumani/lab/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:347
I0226 19:37:56.958740 1 reflector.go:210] Starting reflector *v1.StatefulSet (1h0m0s) from /home/djumani/lab/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:356
I0226 19:37:57.044500 1 reflector.go:246] Listing and watching *v1.StatefulSet from /home/djumani/lab/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:356
I0226 19:37:57.346841 1 registry.go:166] Registering SelectorSpread plugin
I0226 19:37:57.346860 1 registry.go:166] Registering SelectorSpread plugin
I0226 19:37:57.355433 1 cloud_provider_builder.go:29] Building cloudstack cloud provider.
I0226 19:37:57.355701 1 client.go:169] NewAPIRequest API request URL:http://103.13.114.141:8080/client/api?apiKey=*&command=listKubernetesClusters&id=1038d730-7444-4fde-bf1b-9f86712943df&response=json&signature=*
I0226 19:37:57.448589 1 reflector.go:210] Starting reflector *v1.StatefulSet (0s) from k8s.io/client-go/informers/factory.go:134
I0226 19:37:57.448616 1 reflector.go:246] Listing and watching *v1.StatefulSet from k8s.io/client-go/informers/factory.go:134
I0226 19:37:57.448934 1 reflector.go:210] Starting reflector *v1beta1.PodDisruptionBudget (0s) from k8s.io/client-go/informers/factory.go:134
I0226 19:37:57.448947 1 reflector.go:246] Listing and watching *v1beta1.PodDisruptionBudget from k8s.io/client-go/informers/factory.go:134
I0226 19:37:57.449213 1 reflector.go:210] Starting reflector *v1.PersistentVolumeClaim (0s) from k8s.io/client-go/informers/factory.go:134
I0226 19:37:57.449239 1 reflector.go:246] Listing and watching *v1.PersistentVolumeClaim from k8s.io/client-go/informers/factory.go:134
I0226 19:37:57.449511 1 reflector.go:210] Starting reflector *v1.Pod (0s) from k8s.io/client-go/informers/factory.go:134
I0226 19:37:57.449524 1 reflector.go:246] Listing and watching *v1.Pod from k8s.io/client-go/informers/factory.go:134
I0226 19:37:57.544676 1 reflector.go:210] Starting reflector *v1.Node (0s) from k8s.io/client-go/informers/factory.go:134
I0226 19:37:57.544720 1 reflector.go:246] Listing and watching *v1.Node from k8s.io/client-go/informers/factory.go:134
I0226 19:37:57.545074 1 reflector.go:210] Starting reflector *v1.Service (0s) from k8s.io/client-go/informers/factory.go:134
I0226 19:37:57.545086 1 reflector.go:246] Listing and watching *v1.Service from k8s.io/client-go/informers/factory.go:134
I0226 19:37:57.545412 1 reflector.go:210] Starting reflector *v1.ReplicaSet (0s) from k8s.io/client-go/informers/factory.go:134
I0226 19:37:57.545424 1 reflector.go:246] Listing and watching *v1.ReplicaSet from k8s.io/client-go/informers/factory.go:134
I0226 19:37:57.545689 1 reflector.go:210] Starting reflector *v1.PersistentVolume (0s) from k8s.io/client-go/informers/factory.go:134
I0226 19:37:57.545701 1 reflector.go:246] Listing and watching *v1.PersistentVolume from k8s.io/client-go/informers/factory.go:134
I0226 19:37:57.545951 1 reflector.go:210] Starting reflector *v1.StorageClass (0s) from k8s.io/client-go/informers/factory.go:134
I0226 19:37:57.545963 1 reflector.go:246] Listing and watching *v1.StorageClass from k8s.io/client-go/informers/factory.go:134
I0226 19:37:57.546269 1 reflector.go:210] Starting reflector *v1.CSINode (0s) from k8s.io/client-go/informers/factory.go:134
I0226 19:37:57.546281 1 reflector.go:246] Listing and watching *v1.CSINode from k8s.io/client-go/informers/factory.go:134
I0226 19:37:57.546558 1 reflector.go:210] Starting reflector *v1.ReplicationController (0s) from k8s.io/client-go/informers/factory.go:134
I0226 19:37:57.546571 1 reflector.go:246] Listing and watching *v1.ReplicationController from k8s.io/client-go/informers/factory.go:134
W0226 19:37:57.547858 1 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W0226 19:37:57.550087 1 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
I0226 19:37:57.644331 1 request.go:581] Throttling request took 97.592619ms, request: GET:https://10.96.0.1:443/api/v1/replicationcontrollers?limit=500&resourceVersion=0
W0226 19:37:57.645374 1 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W0226 19:37:57.750211 1 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
I0226 19:37:57.844566 1 client.go:175] NewAPIRequest response status code:200
I0226 19:37:57.845813 1 cloudstack_manager.go:88] Got cluster : &{1038d730-7444-4fde-bf1b-9f86712943df maroon 0 0 1 1 [0xc00112a180 0xc00112a1b0]}
I0226 19:37:57.846030 1 main.go:280] Registered cleanup signal handler
I0226 19:37:57.846150 1 node_instances_cache.go:156] Start refreshing cloud provider node instances cache
I0226 19:37:57.846239 1 node_instances_cache.go:168] Refresh cloud provider node instances cache finished, refresh took 4.652µs
I0226 19:38:07.846246 1 static_autoscaler.go:229] Starting main loop
I0226 19:38:07.846488 1 client.go:169] NewAPIRequest API request URL:http://103.13.114.141:8080/client/api?apiKey=*&command=listKubernetesClusters&id=1038d730-7444-4fde-bf1b-9f86712943df&response=json&signature=*
I0226 19:38:07.881961 1 client.go:175] NewAPIRequest response status code:200
I0226 19:38:07.882417 1 cloudstack_manager.go:88] Got cluster : &{1038d730-7444-4fde-bf1b-9f86712943df maroon 0 0 1 1 [0xc000568cc0 0xc000569230]}
E0226 19:38:07.882509 1 static_autoscaler.go:271] Failed to get node infos for groups: Unable to find node maroon-control-17f362f2d09 in cluster
I0226 19:38:17.882782 1 static_autoscaler.go:229] Starting main loop
I0226 19:38:17.882968 1 client.go:169] NewAPIRequest API request URL:http://103.13.114.141:8080/client/api?apiKey=*&command=listKubernetesClusters&id=1038d730-7444-4fde-bf1b-9f86712943df&response=json&signature=*
I0226 19:38:17.915704 1 client.go:175] NewAPIRequest response status code:200
I0226 19:38:17.916052 1 cloudstack_manager.go:88] Got cluster : &{1038d730-7444-4fde-bf1b-9f86712943df maroon 0 0 1 1 [0xc0010b6510 0xc0010b6540]}
E0226 19:38:17.916262 1 static_autoscaler.go:271] Failed to get node infos for groups: Unable to find node maroon-control-17f362f2d09 in cluster
I0226 19:38:27.916535 1 static_autoscaler.go:229] Starting main loop
I0226 19:38:27.916742 1 client.go:169] NewAPIRequest API request URL:http://103.13.114.141:8080/client/api?apiKey=*&command=listKubernetesClusters&id=1038d730-7444-4fde-bf1b-9f86712943df&response=json&signature=*
I0226 19:38:27.949574 1 client.go:175] NewAPIRequest response status code:200
I0226 19:38:27.949944 1 cloudstack_manager.go:88] Got cluster : &{1038d730-7444-4fde-bf1b-9f86712943df maroon 0 0 1 1 [0xc0010de2d0 0xc0010de300]}
E0226 19:38:27.950067 1 static_autoscaler.go:271] Failed to get node infos for groups: Unable to find node maroon-control-17f362f2d09 in cluster
I0226 19:38:37.950294 1 static_autoscaler.go:229] Starting main loop
I0226 19:38:37.950724 1 client.go:169] NewAPIRequest API request URL:http://103.13.114.141:8080/client/api?apiKey=*&command=listKubernetesClusters&id=1038d730-7444-4fde-bf1b-9f86712943df&response=json&signature=*
I0226 19:38:37.986748 1 client.go:175] NewAPIRequest response status code:200
I0226 19:38:37.988575 1 cloudstack_manager.go:88] Got cluster : &{1038d730-7444-4fde-bf1b-9f86712943df maroon 0 0 1 1 [0xc000c74d20 0xc000c74d50]}
E0226 19:38:37.988820 1 static_autoscaler.go:271] Failed to get node infos for groups: Unable to find node maroon-node-17f362f60c4 in cluster
I0226 19:38:47.990150 1 static_autoscaler.go:229] Starting main loop
I0226 19:38:47.990450 1 client.go:169] NewAPIRequest API request URL:http://103.13.114.141:8080/client/api?apiKey=*&command=listKubernetesClusters&id=1038d730-7444-4fde-bf1b-9f86712943df&response=json&signature=*
I0226 19:38:48.025570 1 client.go:175] NewAPIRequest response status code:200
I0226 19:38:48.025909 1 cloudstack_manager.go:88] Got cluster : &{1038d730-7444-4fde-bf1b-9f86712943df maroon 0 0 1 1 [0xc00112bb60 0xc00112bb90]}
E0226 19:38:48.025983 1 static_autoscaler.go:271] Failed to get node infos for groups: Unable to find node maroon-control-17f362f2d09 in cluster
I0226 19:38:58.026181 1 static_autoscaler.go:229] Starting main loop
I0226 19:38:58.026322 1 client.go:169] NewAPIRequest API request URL:http://103.13.114.141:8080/client/api?apiKey=*&command=listKubernetesClusters&id=1038d730-7444-4fde-bf1b-9f86712943df&response=json&signature=*
I0226 19:38:58.059232 1 client.go:175] NewAPIRequest response status code:200
I0226 19:38:58.059452 1 cloudstack_manager.go:88] Got cluster : &{1038d730-7444-4fde-bf1b-9f86712943df maroon 0 0 1 1 [0xc000b83800 0xc000b83830]}
E0226 19:38:58.059515 1 static_autoscaler.go:271] Failed to get node infos for groups: Unable to find node maroon-control-17f362f2d09 in cluster
I0226 19:39:08.059645 1 static_autoscaler.go:229] Starting main loop
I0226 19:39:08.059759 1 client.go:169] NewAPIRequest API request URL:http://103.13.114.141:8080/client/api?apiKey=*&command=listKubernetesClusters&id=1038d730-7444-4fde-bf1b-9f86712943df&response=json&signature=*
I0226 19:39:08.097115 1 client.go:175] NewAPIRequest response status code:200
I0226 19:39:08.097587 1 cloudstack_manager.go:88] Got cluster : &{1038d730-7444-4fde-bf1b-9f86712943df maroon 0 0 1 1 [0xc000b34600 0xc000b34630]}
E0226 19:39:08.097712 1 static_autoscaler.go:271] Failed to get node infos for groups: Unable to find node maroon-control-17f362f2d09 in cluster
I0226 19:39:18.097917 1 static_autoscaler.go:229] Starting main loop
I0226 19:39:18.098085 1 client.go:169] NewAPIRequest API request URL:http://103.13.114.141:8080/client/api?apiKey=*&command=listKubernetesClusters&id=1038d730-7444-4fde-bf1b-9f86712943df&response=json&signature=*
I0226 19:39:18.135679 1 client.go:175] NewAPIRequest response status code:200
I0226 19:39:18.136097 1 cloudstack_manager.go:88] Got cluster : &{1038d730-7444-4fde-bf1b-9f86712943df maroon 0 0 1 1 [0xc000bb2330 0xc000bb2390]}
E0226 19:39:18.136211 1 static_autoscaler.go:271] Failed to get node infos for groups: Unable to find node maroon-node-17f362f60c4 in cluster
I0226 19:39:28.136375 1 static_autoscaler.go:229] Starting main loop
I0226 19:39:28.136557 1 client.go:169] NewAPIRequest API request URL:http://103.13.114.141:8080/client/api?apiKey=*&command=listKubernetesClusters&id=1038d730-7444-4fde-bf1b-9f86712943df&response=json&signature=*
I0226 19:39:28.170063 1 client.go:175] NewAPIRequest response status code:200
I0226 19:39:28.170688 1 cloudstack_manager.go:88] Got cluster : &{1038d730-7444-4fde-bf1b-9f86712943df maroon 0 0 1 1 [0xc000b5c1e0 0xc000b5c210]}
E0226 19:39:28.170871 1 static_autoscaler.go:271] Failed to get node infos for groups: Unable to find node maroon-control-17f362f2d09 in cluster
I0226 19:39:38.171127 1 static_autoscaler.go:229] Starting main loop
I0226 19:39:38.171272 1 client.go:169] NewAPIRequest API request URL:http://103.13.114.141:8080/client/api?apiKey=*&command=listKubernetesClusters&id=1038d730-7444-4fde-bf1b-9f86712943df&response=json&signature=*
I0226 19:39:38.204127 1 client.go:175] NewAPIRequest response status code:200
I0226 19:39:38.204608 1 cloudstack_manager.go:88] Got cluster : &{1038d730-7444-4fde-bf1b-9f86712943df maroon 0 0 1 1 [0xc000b53530 0xc000b53560]}
E0226 19:39:38.204739 1 static_autoscaler.go:271] Failed to get node infos for groups: Unable to find node maroon-control-17f362f2d09 in cluster
I0226 19:39:48.204940 1 static_autoscaler.go:229] Starting main loop
I0226 19:39:48.205066 1 client.go:169] NewAPIRequest API request URL:http://103.13.114.141:8080/client/api?apiKey=*&command=listKubernetesClusters&id=1038d730-7444-4fde-bf1b-9f86712943df&response=json&signature=*
I0226 19:39:48.235646 1 client.go:175] NewAPIRequest response status code:200
I0226 19:39:48.236045 1 cloudstack_manager.go:88] Got cluster : &{1038d730-7444-4fde-bf1b-9f86712943df maroon 0 0 1 1 [0xc000bbb4d0 0xc000bbb500]}
E0226 19:39:48.236109 1 static_autoscaler.go:271] Failed to get node infos for groups: Unable to find node maroon-control-17f362f2d09 in cluster
I0226 19:39:57.846437 1 node_instances_cache.go:156] Start refreshing cloud provider node instances cache
I0226 19:39:57.846502 1 node_instances_cache.go:168] Refresh cloud provider node instances cache finished, refresh took 5.778µs
I0226 19:39:58.236278 1 static_autoscaler.go:229] Starting main loop
I0226 19:39:58.236397 1 client.go:169] NewAPIRequest API request URL:http://103.13.114.141:8080/client/api?apiKey=*&command=listKubernetesClusters&id=1038d730-7444-4fde-bf1b-9f86712943df&response=json&signature=*
I0226 19:39:58.270179 1 client.go:175] NewAPIRequest response status code:200
I0226 19:39:58.270559 1 cloudstack_manager.go:88] Got cluster : &{1038d730-7444-4fde-bf1b-9f86712943df maroon 0 0 1 1 [0xc000eb2f30 0xc000eb2f60]}
E0226 19:39:58.270726 1 static_autoscaler.go:271] Failed to get node infos for groups: Unable to find node maroon-node-17f362f60c4 in cluster
I0226 19:40:08.271034 1 static_autoscaler.go:229] Starting main loop
I0226 19:40:08.271154 1 client.go:169] NewAPIRequest API request URL:http://103.13.114.141:8080/client/api?apiKey=*&command=listKubernetesClusters&id=1038d730-7444-4fde-bf1b-9f86712943df&response=json&signature=*
I0226 19:40:08.303001 1 client.go:175] NewAPIRequest response status code:200
I0226 19:40:08.303372 1 cloudstack_manager.go:88] Got cluster : &{1038d730-7444-4fde-bf1b-9f86712943df maroon 0 0 1 1 [0xc000f7a780 0xc000f7a7b0]}
E0226 19:40:08.303450 1 static_autoscaler.go:271] Failed to get node infos for groups: Unable to find node maroon-control-17f362f2d09 in cluster
I0226 19:40:18.303605 1 static_autoscaler.go:229] Starting main loop
I0226 19:40:18.303732 1 client.go:169] NewAPIRequest API request URL:http://103.13.114.141:8080/client/api?apiKey=*&command=listKubernetesClusters&id=1038d730-7444-4fde-bf1b-9f86712943df&response=json&signature=*
I0226 19:40:18.339958 1 client.go:175] NewAPIRequest response status code:200
I0226 19:40:18.340366 1 cloudstack_manager.go:88] Got cluster : &{1038d730-7444-4fde-bf1b-9f86712943df maroon 0 0 1 1 [0xc001088b40 0xc001088b70]}
E0226 19:40:18.340562 1 static_autoscaler.go:271] Failed to get node infos for groups: Unable to find node maroon-control-17f362f2d09 in cluster
I0226 19:40:28.340915 1 static_autoscaler.go:229] Starting main loop
I0226 19:40:28.341042 1 client.go:169] NewAPIRequest API request URL:http://103.13.114.141:8080/client/api?apiKey=*&command=listKubernetesClusters&id=1038d730-7444-4fde-bf1b-9f86712943df&response=json&signature=*
I0226 19:40:28.377760 1 client.go:175] NewAPIRequest response status code:200
I0226 19:40:28.378288 1 cloudstack_manager.go:88] Got cluster : &{1038d730-7444-4fde-bf1b-9f86712943df maroon 0 0 1 1 [0xc001096e10 0xc001096e40]}
E0226 19:40:28.378496 1 static_autoscaler.go:271] Failed to get node infos for groups: Unable to find node maroon-control-17f362f2d09 in cluster
I0226 19:40:38.378731 1 static_autoscaler.go:229] Starting main loop
I0226 19:40:38.378897 1 client.go:169] NewAPIRequest API request URL:http://103.13.114.141:8080/client/api?apiKey=*&command=listKubernetesClusters&id=1038d730-7444-4fde-bf1b-9f86712943df&response=json&signature=*
I0226 19:40:38.410575 1 client.go:175] NewAPIRequest response status code:200
I0226 19:40:38.410884 1 cloudstack_manager.go:88] Got cluster : &{1038d730-7444-4fde-bf1b-9f86712943df maroon 0 0 1 1 [0xc000ffac90 0xc000ffacc0]}
E0226 19:40:38.410944 1 static_autoscaler.go:271] Failed to get node infos for groups: Unable to find node maroon-control-17f362f2d09 in cluster
I0226 19:40:48.411112 1 static_autoscaler.go:229] Starting main loop
I0226 19:40:48.411270 1 client.go:169] NewAPIRequest API request URL:http://103.13.114.141:8080/client/api?apiKey=*&command=listKubernetesClusters&id=1038d730-7444-4fde-bf1b-9f86712943df&response=json&signature=*
I0226 19:40:48.450260 1 client.go:175] NewAPIRequest response status code:200
I0226 19:40:48.450577 1 cloudstack_manager.go:88] Got cluster : &{1038d730-7444-4fde-bf1b-9f86712943df maroon 0 0 1 1 [0xc0010bf2f0 0xc0010bf320]}
E0226 19:40:48.450672 1 static_autoscaler.go:271] Failed to get node infos for groups: Unable to find node maroon-control-17f362f2d09 in cluster
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] davidjumani edited a comment on issue #5976: Kubernetes Auto Scaling not working in version 4.16.0.0
Posted by GitBox <gi...@apache.org>.
davidjumani edited a comment on issue #5976:
URL: https://github.com/apache/cloudstack/issues/5976#issuecomment-1035881805
@satishglondhe Can you delete the autoscaler pod and let it come up again ? It's sometimes unable to find the node by ID such as `Unable to find node 021334d4-073d-4adf-80ee-16036ae29a1c in cluster`, but finds it by name. If the issue still persists, can you recreate the cluster with another name and check ?
Can you also provide the autoscaler logs for one entire main loop ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] sureshanaparti commented on issue #5976: Kubernetes Auto Scaling not working in version 4.16.0.0
Posted by GitBox <gi...@apache.org>.
sureshanaparti commented on issue #5976:
URL: https://github.com/apache/cloudstack/issues/5976#issuecomment-1034557455
@davidjumani can you check this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] davidjumani edited a comment on issue #5976: Kubernetes Auto Scaling not working in version 4.16.0.0
Posted by GitBox <gi...@apache.org>.
davidjumani edited a comment on issue #5976:
URL: https://github.com/apache/cloudstack/issues/5976#issuecomment-1047423202
Can you edit the deployment and point the container image to `apache/cloudstack-kubernetes-autoscaler:v1.0.0`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] davidjumani commented on issue #5976: Kubernetes Auto Scaling not working in version 4.16.0.0
Posted by GitBox <gi...@apache.org>.
davidjumani commented on issue #5976:
URL: https://github.com/apache/cloudstack/issues/5976#issuecomment-1038973161
@satishglondhe Can you describe the autoscaler deployment? And also set the imagePullPolicy to Always and then destroy the pod ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] satishglondhe commented on issue #5976: Kubernetes Auto Scaling not working in version 4.16.0.0
Posted by GitBox <gi...@apache.org>.
satishglondhe commented on issue #5976:
URL: https://github.com/apache/cloudstack/issues/5976#issuecomment-1037278061
I recreated the cluster with another name, tried deleting the cluster-autoscaler pod, restarted the deployment, but same issue.
Log is
I0211 17:06:40.084817 1 static_autoscaler.go:229] Starting main loop
I0211 17:06:40.085114 1 client.go:169] NewAPIRequest API request URL:http://103.13.114.141:8080/client/api?apiKey=*&command=listKubernetesClusters&id=9964846a-8bf6-4165-a834-c89c95432609&response=json&signature=*
I0211 17:06:40.117925 1 client.go:175] NewAPIRequest response status code:200
I0211 17:06:40.118811 1 cloudstack_manager.go:88] Got cluster : &{9964846a-8bf6-4165-a834-c89c95432609 blue 1 2 1 1 [0xc000a5eae0 0xc000a5eb10]}
E0211 17:06:40.118909 1 static_autoscaler.go:271] Failed to get node infos for groups: Unable to find node blue-control-17ee87508b0 in cluster
I0211 17:06:50.119437 1 static_autoscaler.go:229] Starting main loop
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] satishglondhe commented on issue #5976: Kubernetes Auto Scaling not working in version 4.16.0.0
Posted by GitBox <gi...@apache.org>.
satishglondhe commented on issue #5976:
URL: https://github.com/apache/cloudstack/issues/5976#issuecomment-1044017218
Hi @davidjumani , I have described the autoscaler deployment below. I checked for the policy, it was already set to 'Always', tried deleting the pods many times. Gives the same "failed to get node infos for groups" every time. I checked for the names of instances in Cloud Stack and in cluster, the names match.
`kubectl describe deployment cluster-autoscaler -n kube-system`
Name: cluster-autoscaler
Namespace: kube-system
CreationTimestamp: Thu, 17 Feb 2022 14:58:39 +0000
Labels: app=cluster-autoscaler
Annotations: deployment.kubernetes.io/revision: 1
Selector: app=cluster-autoscaler
Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app=cluster-autoscaler
Annotations: prometheus.io/port: 8085
prometheus.io/scrape: true
Service Account: cluster-autoscaler
Containers:
cluster-autoscaler:
Image: shapeblue/cloudstack-k8s-autoscaler:v1.0.0
Port: <none>
Host Port: <none>
Command:
./cluster-autoscaler
--v=4
--stderrthreshold=info
--cloud-provider=cloudstack
--cloud-config=/config/cloud-config
--skip-nodes-with-local-storage=false
--nodes=1:2:fd55e9d4-474b-49c1-9d96-7721f78765ac
Limits:
cpu: 100m
memory: 300Mi
Requests:
cpu: 100m
memory: 300Mi
Environment: <none>
Mounts:
/config from cloud-config (ro)
Volumes:
cloud-config:
Type: Secret (a volume populated by a Secret)
SecretName: cloudstack-secret
Optional: false
Conditions:
Type Status Reason
---- ------ ------
Progressing True NewReplicaSetAvailable
Available True MinimumReplicasAvailable
OldReplicaSets: <none>
NewReplicaSet: cluster-autoscaler-6cc66979cd (1/1 replicas created)
Events: <none>
`kubectl get nodes -o wide`
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
aqua-control-17f08219ac4 Ready control-plane,master 15h v1.22.2 10.1.1.105 <none> Debian GNU/Linux 11 (bullseye) 5.10.0-8-amd64 docker://20.10.8
aqua-node-17f0821c72b Ready <none> 15h v1.22.2 10.1.1.209 <none> Debian GNU/Linux 11 (bullseye) 5.10.0-8-amd64 docker://20.10.8
Instance Names in Cloudstack UI
aqua-control-17f08219ac4
aqua-node-17f0821c72b
I0218 06:45:36.546531 1 static_autoscaler.go:229] Starting main loop
I0218 06:45:36.546610 1 client.go:169] NewAPIRequest API request URL:http://103.13.114.141:8080/client/api?apiKey=*&command=listKubernetesClusters&id=fd55e9d4-474b-49c1-9d96-7721f78765ac&response=json&signature=*
I0218 06:45:36.580456 1 client.go:175] NewAPIRequest response status code:200
I0218 06:45:36.580868 1 cloudstack_manager.go:88] Got cluster : &{fd55e9d4-474b-49c1-9d96-7721f78765ac aqua 1 2 1 1 [0xc000eda840 0xc000eda870]}
E0218 06:45:36.581008 1 static_autoscaler.go:271] Failed to get node infos for groups: Unable to find node aqua-node-17f0821c72b in cluster
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] davidjumani commented on issue #5976: Kubernetes Auto Scaling not working in version 4.16.0.0
Posted by GitBox <gi...@apache.org>.
davidjumani commented on issue #5976:
URL: https://github.com/apache/cloudstack/issues/5976#issuecomment-1066514021
@satishglondhe Looks to be an issue between the UUID on the hypervisor vs CloudStack. I've made some changes to the autoscaler image. Can you destroy the container, pull and try again ?
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] satishglondhe commented on issue #5976: Kubernetes Auto Scaling not working in version 4.16.0.0
Posted by GitBox <gi...@apache.org>.
satishglondhe commented on issue #5976:
URL: https://github.com/apache/cloudstack/issues/5976#issuecomment-1073432295
@davidjumani the cluster autoscaler is now working as expected. Thank you for helping us through. Just one question, what is the way to replace the default autoscaler image in k8s iso?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] satishglondhe commented on issue #5976: Kubernetes Auto Scaling not working in version 4.16.0.0
Posted by GitBox <gi...@apache.org>.
satishglondhe commented on issue #5976:
URL: https://github.com/apache/cloudstack/issues/5976#issuecomment-1047575485
I changed the image in deployment with the image you mentioned. Same result: Failed to get node infos for groups: Unable to find node in cluster. Following is the deployment.
`kubectl describe deployment cluster-autoscaler -n kube-system`
Name: cluster-autoscaler
Namespace: kube-system
CreationTimestamp: Tue, 22 Feb 2022 08:55:48 +0000
Labels: app=cluster-autoscaler
Annotations: deployment.kubernetes.io/revision: 3
Selector: app=cluster-autoscaler
Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app=cluster-autoscaler
Annotations: kubectl.kubernetes.io/restartedAt: 2022-02-22T08:59:16Z
prometheus.io/port: 8085
prometheus.io/scrape: true
Service Account: cluster-autoscaler
Containers:
cluster-autoscaler:
Image: apache/cloudstack-kubernetes-autoscaler:v1.0.0
Port: <none>
Host Port: <none>
Command:
./cluster-autoscaler
--v=4
--stderrthreshold=info
--cloud-provider=cloudstack
--cloud-config=/config/cloud-config
--skip-nodes-with-local-storage=false
--nodes=1:2:68436724-8ce6-4f7a-b650-0a8982d746f9
Limits:
cpu: 100m
memory: 300Mi
Requests:
cpu: 100m
memory: 300Mi
Environment: <none>
Mounts:
/config from cloud-config (ro)
Volumes:
cloud-config:
Type: Secret (a volume populated by a Secret)
SecretName: cloudstack-secret
Optional: false
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing True NewReplicaSetAvailable
OldReplicaSets: <none>
NewReplicaSet: cluster-autoscaler-bc47f77f7 (1/1 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 5m51s deployment-controller Scaled up replica set cluster-autoscaler-bbbf6f87c to 1
Normal ScalingReplicaSet 3m45s deployment-controller Scaled up replica set cluster-autoscaler-79685fd994 to 1
Normal ScalingReplicaSet 3m39s deployment-controller Scaled down replica set cluster-autoscaler-bbbf6f87c to 0
Normal ScalingReplicaSet 2m22s deployment-controller Scaled up replica set cluster-autoscaler-bc47f77f7 to 1
Normal ScalingReplicaSet 2m17s deployment-controller Scaled down replica set cluster-autoscaler-79685fd994 to 0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] satishglondhe commented on issue #5976: Kubernetes Auto Scaling not working in version 4.16.0.0
Posted by GitBox <gi...@apache.org>.
satishglondhe commented on issue #5976:
URL: https://github.com/apache/cloudstack/issues/5976#issuecomment-1066448292
Hi @davidjumani, Thank you for helping us through. I created a cluster with k8s versions 1.22.6 and 1.23.3, enabled autoscaling, changed the image to davidjumani/cloudstack-kubernetes-autoscaler:name-dev , deployed metrics server, and then created hpa, increased the load on a deployment, I was able to see the cluster scaled from node size 1 to 2. The pending pods were deployed on this new node. The logs are as follows.
```
I0313 15:45:26.022450 1 cloudstack_manager.go:88] Got cluster : &{7c06e610-145c-4806-be2b-99f37a2b2b75 voyager 1 3 1 1 [0xc000d6d2f0 0xc000d6d320]}
W0313 15:45:26.022732 1 clusterstate.go:590] Failed to get nodegroup for 52b47c11-175a-441c-a5a4-96e83971f32a: Unable to find node 52b47c11-175a-441c-a5a4-96e83971f32a in cluster
W0313 15:45:26.022765 1 clusterstate.go:590] Failed to get nodegroup for 4cbcb62e-99ca-4d00-af22-e1eaae2ae691: Unable to find node 4cbcb62e-99ca-4d00-af22-e1eaae2ae691 in cluster
I0313 15:45:26.022794 1 static_autoscaler.go:341] 2 unregistered nodes present
I0313 15:45:26.022849 1 filter_out_schedulable.go:65] Filtering out schedulables
I0313 15:45:26.022867 1 filter_out_schedulable.go:137] Filtered out 0 pods using hints
I0313 15:45:26.022902 1 filter_out_schedulable.go:175] 0 pods were kept as unschedulable based on caching
I0313 15:45:26.022925 1 filter_out_schedulable.go:176] 0 pods marked as unschedulable can be scheduled.
I0313 15:45:26.023018 1 filter_out_schedulable.go:87] No schedulable pods
I0313 15:45:26.023105 1 klogx.go:86] Pod default/php-apache-7656945b6b-nm8b6 is unschedulable
I0313 15:45:26.023142 1 scale_up.go:376] Upcoming 0 nodes
I0313 15:45:26.023296 1 scale_up.go:468] Best option to resize: 7c06e610-145c-4806-be2b-99f37a2b2b75
I0313 15:45:26.024052 1 scale_up.go:472] Estimated 1 nodes needed in 7c06e610-145c-4806-be2b-99f37a2b2b75
I0313 15:45:26.024215 1 scale_up.go:595] Final scale-up plan: [{7c06e610-145c-4806-be2b-99f37a2b2b75 1->2 (max: 3)}]
I0313 15:45:26.024263 1 scale_up.go:691] Scale-up: setting group 7c06e610-145c-4806-be2b-99f37a2b2b75 size to 2
I0313 15:45:26.024890 1 event_sink_logging_wrapper.go:48] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"kube-system", Name:"cluster-autoscaler-status", UID:"d56cd0a7-29fb-4347-b477-f12daac848df", APIVersion:"v1", ResourceVersion:"8822", FieldPath:""}): type: 'Normal' reason: 'ScaledUpGroup' Scale-up: setting group 7c06e610-145c-4806-be2b-99f37a2b2b75 size to 2 instead of 1 (max: 3)
I0313 15:45:26.028139 1 cloudstack_node_group.go:57] Increase Cluster : 7c06e610-145c-4806-be2b-99f37a2b2b75 by 1
I0313 15:45:26.028330 1 client.go:169] NewAPIRequest API request URL:http://103.13.114.141:8080/client/api?apiKey=***&command=scaleKubernetesCluster&id=7c06e610-145c-4806-be2b-99f37a2b2b75&response=json&size=2&signature=***
I0313 15:45:26.056998 1 client.go:175] NewAPIRequest response status code:200
I0313 15:45:36.057714 1 client.go:169] NewAPIRequest API request URL:http://103.13.114.141:8080/client/api?apiKey=***&command=queryAsyncJobResult&jobid=eb1f348f-5a56-4543-b842-bd168da8192a&response=json&signature=***
I0313 15:45:36.076154 1 client.go:175] NewAPIRequest response status code:200
I0313 15:45:36.078394 1 client.go:110] Still waiting for job eb1f348f-5a56-4543-b842-bd168da8192a to complete
I0313 15:45:43.085178 1 reflector.go:536] /home/djumani/lab/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:309: Watch close - *v1beta1.PodDisruptionBudget total 0 items received
W0313 15:45:43.097076 1 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
I0313 15:45:46.058218 1 client.go:169] NewAPIRequest API request URL:http://103.13.114.141:8080/client/api?apiKey=***&command=queryAsyncJobResult&jobid=eb1f348f-5a56-4543-b842-bd168da8192a&response=json&signature=***
I0313 15:45:46.076218 1 client.go:175] NewAPIRequest response status code:200
I0313 15:45:46.076674 1 client.go:110] Still waiting for job eb1f348f-5a56-4543-b842-bd168da8192a to complete
I0313 15:45:49.284991 1 reflector.go:255] Listing and watching *v1beta1.CSIStorageCapacity from [k8s.io/client-go/informers/factory.go:134](http://k8s.io/client-go/informers/factory.go:134)
W0313 15:45:49.293329 1 reflector.go:324] [k8s.io/client-go/informers/factory.go:134](http://k8s.io/client-go/informers/factory.go:134): failed to list *v1beta1.CSIStorageCapacity: [csistoragecapacities.storage.k8s.io](http://csistoragecapacities.storage.k8s.io/) is forbidden: User "system:serviceaccount:kube-system:cluster-autoscaler" cannot list resource "csistoragecapacities" in API group "[storage.k8s.io](http://storage.k8s.io/)" at the cluster scope
E0313 15:45:49.293574 1 reflector.go:138] [k8s.io/client-go/informers/factory.go:134](http://k8s.io/client-go/informers/factory.go:134): Failed to watch *v1beta1.CSIStorageCapacity: failed to list *v1beta1.CSIStorageCapacity: [csistoragecapacities.storage.k8s.io](http://csistoragecapacities.storage.k8s.io/) is forbidden: User "system:serviceaccount:kube-system:cluster-autoscaler" cannot list resource "csistoragecapacities" in API group "[storage.k8s.io](http://storage.k8s.io/)" at the cluster scope
I0313 15:45:53.108193 1 reflector.go:536] /home/djumani/lab/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:356: Watch close - *v1.StatefulSet total 0 items received
I0313 15:45:56.058337 1 client.go:169] NewAPIRequest API request URL:http://103.13.114.141:8080/client/api?apiKey=***&command=queryAsyncJobResult&jobid=eb1f348f-5a56-4543-b842-bd168da8192a&response=json&signature=***
I0313 15:45:56.077064 1 client.go:175] NewAPIRequest response status code:200
```
However, when I removed the load, the autoscaler was able to detect that the new node is not needed based on cpu metrics, then it moved the pods on new node to the old node. When it came to deleting the new node, the operation failed. Following are the logs for same.
```
I0313 16:10:42.751483 1 scale_down.go:418] Skipping voyager-control-17f83bf2095 from delete consideration - the node is marked as no scale down
I0313 16:10:42.751496 1 scale_down.go:444] Node voyager-node-17f83bf431c is not suitable for removal - memory utilization too big (0.519572)
I0313 16:10:42.751526 1 scale_down.go:448] Node voyager-node-17f83f4b273 - cpu utilization 0.150000
I0313 16:10:42.751546 1 cluster.go:139] voyager-node-17f83f4b273 for removal
I0313 16:10:42.751581 1 cluster.go:322] Pod default/php-apache-7656945b6b-nm8b6 can be moved to voyager-node-17f83bf431c
I0313 16:10:42.751594 1 cluster.go:167] node voyager-node-17f83f4b273 may be removed
I0313 16:10:42.751646 1 static_autoscaler.go:523] voyager-node-17f83f4b273 is unneeded since 2022-03-13 16:00:39.61791118 +0000 UTC m=+1238.053807358 duration 10m3.096154674s
I0313 16:10:42.751677 1 static_autoscaler.go:534] Scale down status: unneededOnly=false lastScaleUpTime=2022-03-13 15:45:25.988144618 +0000 UTC m=+324.424040837 lastScaleDownDeleteTime=2022-03-13 14:40:24.439401321 +0000 UTC m=-3577.124702365 lastScaleDownFailTime=2022-03-13 14:40:24.439401321 +0000 UTC m=-3577.124702365 scaleDownForbidden=false isDeleteInProgress=false scaleDownInCooldown=false
I0313 16:10:42.751693 1 static_autoscaler.go:547] Starting scale down
I0313 16:10:42.751715 1 scale_down.go:828] voyager-node-17f83f4b273 was unneeded for 10m3.096154674s
I0313 16:10:42.751731 1 cluster.go:139] voyager-node-17f83f4b273 for removal
I0313 16:10:42.751768 1 cluster.go:322] Pod default/php-apache-7656945b6b-nm8b6 can be moved to voyager-node-17f83bf431c
I0313 16:10:42.751781 1 cluster.go:167] node voyager-node-17f83f4b273 may be removed
I0313 16:10:42.751804 1 scale_down.go:980] Scale-down: removing node voyager-node-17f83f4b273, utilization: {0.15 0.1082440988995888 0 cpu 0.15}, pods to reschedule: default/php-apache-7656945b6b-nm8b6
I0313 16:10:42.752137 1 event_sink_logging_wrapper.go:48] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"kube-system", Name:"cluster-autoscaler-status", UID:"d56cd0a7-29fb-4347-b477-f12daac848df", APIVersion:"v1", ResourceVersion:"13552", FieldPath:""}): type: 'Normal' reason: 'ScaleDown' Scale-down: removing node voyager-node-17f83f4b273, utilization: {0.15 0.1082440988995888 0 cpu 0.15}, pods to reschedule: default/php-apache-7656945b6b-nm8b6
I0313 16:10:42.763508 1 delete.go:103] Successfully added ToBeDeletedTaint on node voyager-node-17f83f4b273
I0313 16:10:42.765309 1 event_sink_logging_wrapper.go:48] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"voyager-node-17f83f4b273", UID:"56023857-197a-4844-ae26-443c541066b0", APIVersion:"v1", ResourceVersion:"12899", FieldPath:""}): type: 'Normal' reason: 'ScaleDown' marked the node as toBeDeleted/unschedulable
I0313 16:10:42.772269 1 event_sink_logging_wrapper.go:48] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"weave-net-qjv74", UID:"526d282a-5c5e-41f9-af1c-5008730dfaae", APIVersion:"v1", ResourceVersion:"9683", FieldPath:""}): type: 'Normal' reason: 'ScaleDown' deleting pod for node scale down
I0313 16:10:42.776486 1 event_sink_logging_wrapper.go:48] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"php-apache-7656945b6b-nm8b6", UID:"b68a74d7-374a-4367-9784-737e3bdd3f80", APIVersion:"v1", ResourceVersion:"9943", FieldPath:""}): type: 'Normal' reason: 'ScaleDown' deleting pod for node scale down
I0313 16:10:42.808539 1 event_sink_logging_wrapper.go:48] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"kube-proxy-shdrg", UID:"33ad9fac-0cf5-495c-9147-062d874f3005", APIVersion:"v1", ResourceVersion:"9660", FieldPath:""}): type: 'Normal' reason: 'ScaleDown' deleting pod for node scale down
E0313 16:10:42.830691 1 scale_down.go:1349] Not deleted yet default/php-apache-7656945b6b-nm8b6
I0313 16:10:45.116519 1 reflector.go:536] /home/djumani/lab/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:356: Watch close - *v1.StatefulSet total 7 items received
I0313 16:10:47.836224 1 scale_down.go:1360] All pods removed from voyager-node-17f83f4b273
I0313 16:10:47.836668 1 client.go:169] NewAPIRequest API request URL:http://103.13.114.141:8080/client/api?apiKey=***&command=scaleKubernetesCluster&id=7c06e610-145c-4806-be2b-99f37a2b2b75&nodeids=b23c7e87-5732-65e0-3d9c-0459c34c2115&response=json&signature=***
I0313 16:10:47.849279 1 client.go:175] NewAPIRequest response status code:431
I0313 16:10:47.851936 1 delete.go:197] Releasing taint {Key:ToBeDeletedByClusterAutoscaler Value:1647187842 Effect:NoSchedule TimeAdded:<nil>} on node voyager-node-17f83f4b273
I0313 16:10:47.860208 1 delete.go:228] Successfully released ToBeDeletedTaint on node voyager-node-17f83f4b273
E0313 16:10:47.860245 1 scale_down.go:1006] Failed to delete voyager-node-17f83f4b273: failed to delete voyager-node-17f83f4b273: Unable to delete [b23c7e87-5732-65e0-3d9c-0459c34c2115] from cluster : (HTTP 431, error code 9999) Unable to execute API command scalekubernetescluster due to invalid value. Invalid parameter nodeids value=b23c7e87-5732-65e0-3d9c-0459c34c2115 due to incorrect long value format, or entity does not exist or due to incorrect parameter annotation for the field in api cmd class.
I0313 16:10:47.860284 1 event_sink_logging_wrapper.go:48] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"voyager-node-17f83f4b273", UID:"56023857-197a-4844-ae26-443c541066b0", APIVersion:"v1", ResourceVersion:"12899", FieldPath:""}): type: 'Warning' reason: 'ScaleDownFailed' failed to delete the node
I0313 16:10:50.050998 1 reflector.go:255] Listing and watching *v1.CSIDriver from [k8s.io/client-go/informers/factory.go:134](http://k8s.io/client-go/informers/factory.go:134)
W0313 16:10:50.055262 1 reflector.go:324] [k8s.io/client-go/informers/factory.go:134](http://k8s.io/client-go/informers/factory.go:134): failed to list *v1.CSIDriver: [csidrivers.storage.k8s.io](http://csidrivers.storage.k8s.io/) is forbidden: User "system:serviceaccount:kube-system:cluster-autoscaler" cannot list resource "csidrivers" in API group "[storage.k8s.io](http://storage.k8s.io/)" at the cluster scope
E0313 16:10:50.055307 1 reflector.go:138] [k8s.io/client-go/informers/factory.go:134](http://k8s.io/client-go/informers/factory.go:134): Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: [csidrivers.storage.k8s.io](http://csidrivers.storage.k8s.io/) is forbidden: User "system:serviceaccount:kube-system:cluster-autoscaler" cannot list resource "csidrivers" in API group "[storage.k8s.io](http://storage.k8s.io/)" at the cluster scope
I0313 16:10:52.766096 1 static_autoscaler.go:235] Starting main loop
I0313 16:10:52.766309 1 client.go:169] NewAPIRequest API request URL:http://103.13.114.141:8080/client/api?apiKey=***&command=listKubernetesClusters&id=7c06e610-145c-4806-be2b-99f37a2b2b75&response=json&signature=***
I0313 16:10:52.802153 1 client.go:175] NewAPIRequest response status code:200
I0313 16:10:52.802615 1 cloudstack_manager.go:88] Got cluster : &{7c06e610-145c-4806-be2b-99f37a2b2b75 voyager 1 3 2 1 [0xc0010993b0 0xc0010993e0 0xc001099410]}
W0313 16:10:52.803058 1 clusterstate.go:590] Failed to get nodegroup for 4cbcb62e-99ca-4d00-af22-e1eaae2ae691: Unable to find node 4cbcb62e-99ca-4d00-af22-e1eaae2ae691 in cluster
W0313 16:10:52.803075 1 clusterstate.go:590] Failed to get nodegroup for 52b47c11-175a-441c-a5a4-96e83971f32a: Unable to find node 52b47c11-175a-441c-a5a4-96e83971f32a in cluster
W0313 16:10:52.803080 1 clusterstate.go:590] Failed to get nodegroup for cec92edd-2c62-450b-8af5-ced062c7a84d: Unable to find node cec92edd-2c62-450b-8af5-ced062c7a84d in cluster
I0313 16:10:52.803099 1 static_autoscaler.go:341] 3 unregistered nodes present
I0313 16:10:52.803108 1 static_autoscaler.go:624] Removing unregistered node cec92edd-2c62-450b-8af5-ced062c7a84d
W0313 16:10:52.803114 1 static_autoscaler.go:627] Failed to get node group for cec92edd-2c62-450b-8af5-ced062c7a84d: Unable to find node cec92edd-2c62-450b-8af5-ced062c7a84d in cluster
W0313 16:10:52.803120 1 static_autoscaler.go:346] Failed to remove unregistered nodes: Unable to find node cec92edd-2c62-450b-8af5-ced062c7a84d in cluster
```
These are logs after 20 mins
```
I0313 16:31:09.238844 1 static_autoscaler.go:547] Starting scale down
I0313 16:31:09.238889 1 scale_down.go:828] voyager-node-17f83f4b273 was unneeded for 10m3.256641915s
I0313 16:31:09.238905 1 scale_down.go:1102] Scale-down: removing empty node voyager-node-17f83f4b273
I0313 16:31:09.239138 1 event_sink_logging_wrapper.go:48] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"kube-system", Name:"cluster-autoscaler-status", UID:"d56cd0a7-29fb-4347-b477-f12daac848df", APIVersion:"v1", ResourceVersion:"17347", FieldPath:""}): type: 'Normal' reason: 'ScaleDownEmpty' Scale-down: removing empty node voyager-node-17f83f4b273
I0313 16:31:09.255891 1 delete.go:103] Successfully added ToBeDeletedTaint on node voyager-node-17f83f4b273
I0313 16:31:09.256402 1 client.go:169] NewAPIRequest API request URL:http://103.13.114.141:8080/client/api?apiKey=***&command=scaleKubernetesCluster&id=7c06e610-145c-4806-be2b-99f37a2b2b75&nodeids=b23c7e87-5732-65e0-3d9c-0459c34c2115&response=json&signature=***
I0313 16:31:09.267422 1 client.go:175] NewAPIRequest response status code:431
E0313 16:31:09.269341 1 scale_down.go:1144] Problem with empty node deletion: failed to delete voyager-node-17f83f4b273: Unable to delete [b23c7e87-5732-65e0-3d9c-0459c34c2115] from cluster : (HTTP 431, error code 9999) Unable to execute API command scalekubernetescluster due to invalid value. Invalid parameter nodeids value=b23c7e87-5732-65e0-3d9c-0459c34c2115 due to incorrect long value format, or entity does not exist or due to incorrect parameter annotation for the field in api cmd class.
I0313 16:31:09.272042 1 delete.go:197] Releasing taint {Key:ToBeDeletedByClusterAutoscaler Value:1647189069 Effect:NoSchedule TimeAdded:<nil>} on node voyager-node-17f83f4b273
I0313 16:31:09.279999 1 delete.go:228] Successfully released ToBeDeletedTaint on node voyager-node-17f83f4b273
I0313 16:31:09.306255 1 event_sink_logging_wrapper.go:48] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"voyager-node-17f83f4b273", UID:"56023857-197a-4844-ae26-443c541066b0", APIVersion:"v1", ResourceVersion:"16596", FieldPath:""}): type: 'Warning' reason: 'ScaleDownFailed' failed to delete empty node: failed to delete voyager-node-17f83f4b273: Unable to delete [b23c7e87-5732-65e0-3d9c-0459c34c2115] from cluster : (HTTP 431, error code 9999) Unable to execute API command scalekubernetescluster due to invalid value. Invalid parameter nodeids value=b23c7e87-5732-65e0-3d9c-0459c34c2115 due to incorrect long value format, or entity does not exist or due to incorrect parameter annotation for the field in api cmd class.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] davidjumani commented on issue #5976: Kubernetes Auto Scaling not working in version 4.16.0.0
Posted by GitBox <gi...@apache.org>.
davidjumani commented on issue #5976:
URL: https://github.com/apache/cloudstack/issues/5976#issuecomment-1074721972
Fixed via https://github.com/kubernetes/autoscaler/commit/9cd06c2f85366400cec498536fc172aa587a13a7
Will rebuild the autoscaler and update the isos
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] davidjumani commented on issue #5976: Kubernetes Auto Scaling not working in version 4.16.0.0
Posted by GitBox <gi...@apache.org>.
davidjumani commented on issue #5976:
URL: https://github.com/apache/cloudstack/issues/5976#issuecomment-1061881801
@satishglondhe Can you try editing the autoscaler deployment and using `davidjumani/cloudstack-kubernetes-autoscaler:name-dev` as the image ? I've made a minor change which might resolve the issue
As for the setup I've used and tested, it's a 1.22 / 1.23 with 1-CP / 1-3WN setup, tested on KVM, VMware and Xen. @vladimirpetrov Might have more info on the specifics of the hypervisors
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] davidjumani commented on issue #5976: Kubernetes Auto Scaling not working in version 4.16.0.0
Posted by GitBox <gi...@apache.org>.
davidjumani commented on issue #5976:
URL: https://github.com/apache/cloudstack/issues/5976#issuecomment-1073633810
Thanks @satishglondhe I'll update the autoscaler image. To replace the image, either directly modify the iso, or rebuild it
https://github.com/apache/cloudstack/blob/main/scripts/util/create-kubernetes-binaries-iso.sh
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] davidjumani commented on issue #5976: Kubernetes Auto Scaling not working in version 4.16.0.0
Posted by GitBox <gi...@apache.org>.
davidjumani commented on issue #5976:
URL: https://github.com/apache/cloudstack/issues/5976#issuecomment-1035881805
@satishglondhe Can you delete the autoscaler pod and let it come up again ? It's sometimes unable to find the node by ID such as `Unable to find node 021334d4-073d-4adf-80ee-16036ae29a1c in cluster`, but finds it by name. If the issue still persists, can you recreate the cluster with another name and check ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] satishglondhe commented on issue #5976: Kubernetes Auto Scaling not working in version 4.16.0.0
Posted by GitBox <gi...@apache.org>.
satishglondhe commented on issue #5976:
URL: https://github.com/apache/cloudstack/issues/5976#issuecomment-1037278061
I recreated the cluster with another name, tried deleting the cluster-autoscaler pod, restarted the deployment, but same issue.
Log is
I0211 17:06:40.084817 1 static_autoscaler.go:229] Starting main loop
I0211 17:06:40.085114 1 client.go:169] NewAPIRequest API request URL:http://103.13.114.141:8080/client/api?apiKey=*&command=listKubernetesClusters&id=9964846a-8bf6-4165-a834-c89c95432609&response=json&signature=*
I0211 17:06:40.117925 1 client.go:175] NewAPIRequest response status code:200
I0211 17:06:40.118811 1 cloudstack_manager.go:88] Got cluster : &{9964846a-8bf6-4165-a834-c89c95432609 blue 1 2 1 1 [0xc000a5eae0 0xc000a5eb10]}
E0211 17:06:40.118909 1 static_autoscaler.go:271] Failed to get node infos for groups: Unable to find node blue-control-17ee87508b0 in cluster
I0211 17:06:50.119437 1 static_autoscaler.go:229] Starting main loop
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org