You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "ugenk (via GitHub)" <gi...@apache.org> on 2024/04/18 19:44:37 UTC
[I] problem on adding new node to kubernetes [cloudstack]
ugenk opened a new issue, #8943:
URL: https://github.com/apache/cloudstack/issues/8943
##### ISSUE TYPE
* Bug Report
##### COMPONENT NAME
~~~
Kubernetes
~~~
##### CLOUDSTACK VERSION
~~~
4.19.0.1
~~~
##### CONFIGURATION
~~~
Network offering: DefaultNetworkOfferingforKubernetesService
~~~
##### OS / ENVIRONMENT
N/A
##### SUMMARY
While resizing k8s cluster, we're getting error related to adding firewall rules.
##### STEPS TO REPRODUCE
We're reproducing that constantly on our installed cluster. Please, share which information do you need to help you to reproduce that.
Here is our firewall and portforwarding rules for that SNAT IP:
~~~
mysql> select * from firewall_rules where ip_address_id = 4433;
+-------+--------------------------------------+---------------+------------+----------+--------+----------+----------------+------------+-----------+------------+--------------------------------------+---------------------+-----------+-----------+---------+------+--------+--------------+---------+
| id | uuid | ip_address_id | start_port | end_port | state | protocol | purpose | account_id | domain_id | network_id | xid | created | icmp_code | icmp_type | related | type | vpc_id | traffic_type | display |
+-------+--------------------------------------+---------------+------------+----------+--------+----------+----------------+------------+-----------+------------+--------------------------------------+---------------------+-----------+-----------+---------+------+--------+--------------+---------+
| 157 | 160544dc-303a-4449-9ee5-9aa7c74ecd8e | 4433 | 6443 | 6443 | Active | tcp | Firewall | 23 | 1 | 267 | 592c3df8-3804-4ccb-8030-f1a16c77e431 | 2023-05-27 08:18:26 | NULL | NULL | NULL | User | NULL | Ingress | 1 |
| 159 | 53a32814-ab9a-413d-9fba-834c60cbe41c | 4433 | 6443 | 6443 | Active | tcp | LoadBalancing | 23 | 1 | 267 | 8bfbb10f-5577-4a22-b20e-19f7a6c155ff | 2023-05-27 08:18:29 | NULL | NULL | NULL | User | NULL | NULL | 1 |
| 169 | 11c33409-788b-4639-9e77-39f124a97cd8 | 4433 | 3222 | 3222 | Active | tcp | PortForwarding | 23 | 1 | 267 | b3c13c77-0d29-4fe1-9d6d-6e023acab91c | 2023-05-29 20:28:54 | NULL | NULL | NULL | User | NULL | NULL | 1 |
| 171 | 97f9a76f-36f1-4f2b-80da-6fedf100edcb | 4433 | 3222 | 3222 | Active | tcp | Firewall | 23 | 1 | 267 | aac0f2b6-488b-4e6b-aff7-47e26dc588f8 | 2023-05-29 20:32:47 | NULL | NULL | NULL | User | NULL | Ingress | 1 |
| 202 | ede86b93-2fab-4662-a75f-a7eaee50c736 | 4433 | 2222 | 2222 | Active | tcp | PortForwarding | 23 | 1 | 267 | e1645b58-8430-42e4-9108-2f474e71dc34 | 2023-05-31 13:12:51 | NULL | NULL | NULL | User | NULL | NULL | 1 |
| 203 | 1ae4c78a-5736-4a14-b22d-5dc4876f3ebc | 4433 | 2223 | 2223 | Active | tcp | PortForwarding | 23 | 1 | 267 | 35d2cbb1-9644-4a2c-ab59-e9de8cf32219 | 2023-05-31 13:12:52 | NULL | NULL | NULL | User | NULL | NULL | 1 |
| 204 | 2ae6750d-5d51-4812-b60f-abdba5c8cc37 | 4433 | 2224 | 2224 | Active | tcp | PortForwarding | 23 | 1 | 267 | d20b18a2-9287-415a-bcd2-69cf4c08414b | 2023-05-31 13:12:54 | NULL | NULL | NULL | User | NULL | NULL | 1 |
| 205 | 315e1cb9-f0d2-438c-92b2-babbb346b8b9 | 4433 | 2225 | 2225 | Active | tcp | PortForwarding | 23 | 1 | 267 | a971ffbe-cf56-4191-8589-8f660116124c | 2023-05-31 13:12:55 | NULL | NULL | NULL | User | NULL | NULL | 1 |
| 206 | 3cc5707a-a5f0-4eaa-98db-31190e4f777a | 4433 | 2226 | 2226 | Active | tcp | PortForwarding | 23 | 1 | 267 | de06f445-9db3-4652-8ef4-6dc82e2f36ef | 2023-05-31 13:12:56 | NULL | NULL | NULL | User | NULL | NULL | 1 |
| 207 | a568da68-a0a8-4eae-9e12-8ed0ebd55a8a | 4433 | 2227 | 2227 | Active | tcp | PortForwarding | 23 | 1 | 267 | 4e7bebda-591c-4051-bab3-a829ee3e888c | 2023-05-31 13:12:57 | NULL | NULL | NULL | User | NULL | NULL | 1 |
| 210 | 878750f8-cdaa-42bf-af6a-46a5d531eab3 | 4433 | 3223 | 3223 | Active | tcp | Firewall | 23 | 1 | 267 | 4a497f27-af34-4ff5-9d8f-77be06cf080f | 2023-05-31 17:10:58 | NULL | NULL | NULL | User | NULL | Ingress | 1 |
| 211 | 40785ce1-ca7d-46f3-ac87-45c52dccd078 | 4433 | 3223 | 3223 | Active | tcp | PortForwarding | 23 | 1 | 267 | 45fff41a-8d5e-446f-9766-03ce2ffc522a | 2023-05-31 17:11:29 | NULL | NULL | NULL | User | NULL | NULL | 1 |
| 259 | 227e9e80-008f-4598-8474-f983d559d355 | 4433 | 3224 | 3224 | Active | tcp | PortForwarding | 23 | 1 | 267 | 3086de53-2d84-49a5-b52d-b62a6ac23943 | 2023-06-21 13:57:34 | NULL | NULL | NULL | User | NULL | NULL | 1 |
| 260 | f93c69ce-598e-4d5d-b9d6-a122bcb937da | 4433 | 3224 | 3224 | Active | tcp | Firewall | 23 | 1 | 267 | b9d22186-6bbd-4eed-a864-97fe3e44698e | 2023-06-21 13:58:08 | NULL | NULL | NULL | User | NULL | Ingress | 1 |
| 281 | 21962db8-97f1-4709-8085-70bfa5cfa47f | 4433 | 35432 | 35432 | Active | tcp | PortForwarding | 23 | 1 | 267 | 5eddfc85-fec9-48b3-b4e6-8f2a44669c43 | 2023-06-23 09:32:01 | NULL | NULL | NULL | User | NULL | NULL | 1 |
| 282 | a2e99cd4-a3e9-493f-a461-d3827b5b8c6d | 4433 | 35432 | 35432 | Active | tcp | Firewall | 23 | 1 | 267 | 416c60de-3b8a-4891-8bd7-729b6ea31fb1 | 2023-06-23 09:39:29 | NULL | NULL | NULL | User | NULL | Ingress | 1 |
| 284 | 8505e874-127b-4523-af26-0c4462608a3f | 4433 | NULL | NULL | Active | icmp | Firewall | 23 | 1 | 267 | 5f2e4b46-91f6-4c1f-a24f-5480cb63d052 | 2023-07-02 10:25:40 | -1 | -1 | NULL | User | NULL | Ingress | 1 |
| 1470 | ae9602cb-ad43-4251-8b1d-9ca0341a6109 | 4433 | 32727 | 32727 | Active | tcp | LoadBalancing | 23 | 1 | 267 | 11ace4c3-7c0e-4ec7-ae22-44dd06e8ce5c | 2023-07-08 13:20:19 | NULL | NULL | NULL | User | NULL | NULL | 1 |
| 1471 | 6a2753e3-3f87-4183-b541-3274dac0c724 | 4433 | 32727 | 32727 | Active | tcp | Firewall | 23 | 1 | 267 | ed955550-d415-4f8d-8460-4983f5b8fb29 | 2023-07-08 13:20:19 | NULL | NULL | NULL | User | NULL | Ingress | 1 |
| 3211 | 144fbdaa-b95e-4a86-ad8b-bc777e87e7c3 | 4433 | 5601 | 5601 | Active | tcp | PortForwarding | 23 | 1 | 267 | f39579fb-8d84-448b-86e9-c77939b42e95 | 2023-07-14 12:19:22 | NULL | NULL | NULL | User | NULL | NULL | 1 |
| 3213 | 1bc0cef8-dace-48f8-b0ae-bda9a9451797 | 4433 | 1 | 65535 | Active | tcp | Firewall | 23 | 1 | 267 | c3bd9b08-9957-481e-a81a-ef470ac744f6 | 2023-07-14 12:21:51 | NULL | NULL | NULL | User | NULL | Ingress | 1 |
| 3214 | b5b6b139-839a-47d0-80da-8f6b775855c4 | 4433 | 1 | 65535 | Active | tcp | Firewall | 23 | 1 | 267 | ef6eb373-b177-4f5b-bc42-3b9604629f67 | 2023-07-14 12:22:11 | NULL | NULL | NULL | User | NULL | Ingress | 1 |
| 3223 | cee5dc3d-a53e-43a1-ab52-cfadfd223382 | 4433 | 1 | 65535 | Active | tcp | Firewall | 23 | 1 | 267 | 0ca97b93-9784-4c5b-b06d-22079340f8dc | 2023-07-14 13:04:01 | NULL | NULL | NULL | User | NULL | Ingress | 1 |
| 15760 | 4fafe230-fc83-4540-ba11-271c0d717c75 | 4433 | 2228 | 2230 | Active | tcp | Firewall | 23 | 1 | 267 | 48f8bf55-cc62-4faa-a95d-bd244dbfcf98 | 2023-09-14 08:25:01 | NULL | NULL | NULL | User | NULL | Ingress | 1 |
| 39225 | 9087fdc2-1f6c-4ec3-902e-a7cd1b7b8b08 | 4433 | 2228 | 2228 | Active | tcp | PortForwarding | 23 | 1 | 267 | 69417616-fd55-40f6-bfcd-14b2a21b6137 | 2023-12-05 10:02:08 | NULL | NULL | NULL | User | NULL | NULL | 1 |
| 48706 | 910ced9a-965e-40cf-94d4-1cb40507e576 | 4433 | 1 | 65535 | Active | tcp | Firewall | 23 | 1 | 267 | 43342855-4472-4686-b44b-7709545163a2 | 2024-01-09 02:40:01 | NULL | NULL | NULL | User | NULL | Ingress | 1 |
| 48707 | 4ed50fde-a0c9-4b4b-8bad-7d6d9eb95948 | 4433 | 2929 | 2929 | Active | tcp | Firewall | 23 | 1 | 267 | f33949bd-bb92-4ead-85df-1b18ae26ad0f | 2024-01-17 03:23:08 | NULL | NULL | NULL | User | NULL | Ingress | 1 |
| 48708 | 8b351e99-4c26-402b-971c-11decac854ff | 4433 | 500 | 500 | Staged | udp | Vpn | 23 | 1 | 267 | bc287688-aedc-4911-a7fb-6911327855c8 | 2024-01-21 06:13:47 | NULL | NULL | NULL | User | NULL | NULL | 1 |
| 48709 | af332d08-8cb7-4d78-b466-fadc001cc26c | 4433 | 500 | 500 | Active | udp | Firewall | 23 | 1 | 267 | 27884745-0c75-455f-9a75-ce6e6d88c325 | 2024-01-21 06:13:47 | NULL | NULL | 48708 | User | NULL | Ingress | 1 |
| 48710 | 6d7c444b-1044-48b6-99b1-a7c125fa479c | 4433 | 1701 | 1701 | Staged | udp | Vpn | 23 | 1 | 267 | 433eaf8d-a020-4899-8aab-1c2777871e5f | 2024-01-21 06:13:47 | NULL | NULL | NULL | User | NULL | NULL | 1 |
| 48711 | 3fa7d224-1608-4125-83e4-7107388ad66c | 4433 | 1701 | 1701 | Active | udp | Firewall | 23 | 1 | 267 | 9b486a5b-a359-47dc-88cc-f760dfd44a2d | 2024-01-21 06:13:47 | NULL | NULL | 48710 | User | NULL | Ingress | 1 |
| 48712 | bd0aa221-32f0-48d5-8f7a-5bbe4e3a8dbe | 4433 | 4500 | 4500 | Staged | udp | Vpn | 23 | 1 | 267 | 580ceedd-d867-41d7-93f2-e50737f9ed97 | 2024-01-21 06:13:47 | NULL | NULL | NULL | User | NULL | NULL | 1 |
| 48713 | b7c9636c-5421-45fc-b779-e600e1c24db8 | 4433 | 4500 | 4500 | Active | udp | Firewall | 23 | 1 | 267 | bfe7a1bc-c862-41a1-bdd6-e9274716310b | 2024-01-21 06:13:47 | NULL | NULL | 48712 | User | NULL | Ingress | 1 |
+-------+--------------------------------------+---------------+------------+----------+--------+----------+----------------+------------+-----------+------------+--------------------------------------+---------------------+-----------+-----------+---------+------+--------+--------------+---------+
33 rows in set (0.00 sec)
##### EXPECTED RESULTS
Fix the problem
##### ACTUAL RESULTS
~~~
2024-04-18 22:07:11,414 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-1:ctx-d42166a9 job-4626/job-4627) (logid:e142ff5b) Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 392, job origin: 4626
2024-04-18 22:07:11,415 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-1:ctx-d42166a9 job-4626/job-4627) (logid:e142ff5b) Done executing com.cloud.vm.VmWorkStart for job-4627
2024-04-18 22:07:11,417 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-1:ctx-d42166a9 job-4626/job-4627) (logid:e142ff5b) Remove job-4627 from job monitoring
2024-04-18 22:07:11,421 INFO [c.c.k.c.a.KubernetesClusterActionWorker] (API-Job-Executor-1:ctx-7a5b66c2 job-4626 ctx-9e48efab) (logid:e142ff5b) Started VM : mediatech-node-18ef29acb54 in the Kubernetes cluster : mediatech
2024-04-18 22:07:11,424 INFO [c.c.k.c.a.KubernetesClusterActionWorker] (API-Job-Executor-1:ctx-7a5b66c2 job-4626 ctx-9e48efab) (logid:e142ff5b) Provisioned node VM : mediatech-node-18ef29acb54 in to the Kubernetes cluster : mediatech
2024-04-18 22:07:11,435 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-1:ctx-7a5b66c2 job-4626) (logid:e142ff5b) Unexpected exception while executing org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd
java.lang.NullPointerException
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterResourceModifierActionWorker.removeSshFirewallRule(KubernetesClusterResourceModifierActionWorker.java:520)
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterScaleWorker.scaleKubernetesClusterIsolatedNetworkRules(KubernetesClusterScaleWorker.java:124)
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterScaleWorker.scaleKubernetesClusterNetworkRules(KubernetesClusterScaleWorker.java:174)
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterScaleWorker.scaleUpKubernetesClusterSize(KubernetesClusterScaleWorker.java:376)
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterScaleWorker.scaleKubernetesClusterSize(KubernetesClusterScaleWorker.java:412)
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterScaleWorker.scaleCluster(KubernetesClusterScaleWorker.java:465)
at com.cloud.kubernetes.cluster.KubernetesClusterManagerImpl.scaleKubernetesCluster(KubernetesClusterManagerImpl.java:1548)
at org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd.execute(ScaleKubernetesClusterCmd.java:156)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2024-04-18 22:07:11,448 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-1:ctx-7a5b66c2 job-4626) (logid:e142ff5b) Complete async job-4626, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530"}
~~~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] problem on adding new node to kubernetes [cloudstack]
Posted by "ugenk (via GitHub)" <gi...@apache.org>.
ugenk commented on issue #8943:
URL: https://github.com/apache/cloudstack/issues/8943#issuecomment-2074664270
we have 4.19.0.1, so we have to wait 4.19.1 release i'm afraid
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] problem on adding new node to kubernetes [cloudstack]
Posted by "rohityadavcloud (via GitHub)" <gi...@apache.org>.
rohityadavcloud closed issue #8943: problem on adding new node to kubernetes
URL: https://github.com/apache/cloudstack/issues/8943
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] problem on adding new node to kubernetes [cloudstack]
Posted by "rohityadavcloud (via GitHub)" <gi...@apache.org>.
rohityadavcloud commented on issue #8943:
URL: https://github.com/apache/cloudstack/issues/8943#issuecomment-2085346556
Fixed in https://github.com/apache/cloudstack/pull/8107
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] problem on adding new node to kubernetes [cloudstack]
Posted by "weizhouapache (via GitHub)" <gi...@apache.org>.
weizhouapache commented on issue #8943:
URL: https://github.com/apache/cloudstack/issues/8943#issuecomment-2074681059
> we have 4.19.0.1, so we have to wait 4.19.1 release i'm afraid
you can cherry-pick the commit to 4.19.0.1, build the packages and deploy it. @ugenk
cloudstack is a open-source project, you can build whenever needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] problem on adding new node to kubernetes [cloudstack]
Posted by "ugenk (via GitHub)" <gi...@apache.org>.
ugenk commented on issue #8943:
URL: https://github.com/apache/cloudstack/issues/8943#issuecomment-2069758061
The exception originates from here:
https://github.com/apache/cloudstack/blob/4.19.0.1/plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterResourceModifierActionWorker.java#L520
Has it been addressed in https://github.com/apache/cloudstack/pull/8107
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] problem on adding new node to kubernetes [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on issue #8943:
URL: https://github.com/apache/cloudstack/issues/8943#issuecomment-2074498406
@ugenk this code has been merged and should be in 4.19.1 as well, Have you tested with 4.18.2 or a newer version of 4.19?
The description indicates that it is fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org