You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2022/10/12 14:42:57 UTC
[GitHub] [apisix-ingress-controller] shareinto opened a new issue, #1378: bug: init stuck case many namespace
shareinto opened a new issue, #1378:
URL: https://github.com/apache/apisix-ingress-controller/issues/1378
### Issue description
my k8s cluster has about 15k namespaces
```
k get ns | wc -l
14846
```
and log dispaly that every namespace stuck 10 seconds, so the provider start took a long time
```
I1012 21:39:58.101271 1 request.go:665] Waited for 1.000058582s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/apisix.apache.org/v2beta3/namespaces/wshop-disagreement42/apisixroutes
[GIN] 2022/10/12 - 21:39:59 | 200 | 66.378µs | 10.0.254.118 | GET "/healthz"
[GIN] 2022/10/12 - 21:40:02 | 200 | 48.148µs | 10.0.254.118 | GET "/healthz"
I1012 21:40:08.101818 1 request.go:665] Waited for 11.000514374s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/apisix.apache.org/v2beta3/namespaces/wshop-negligee64/apisixroutes
[GIN] 2022/10/12 - 21:40:09 | 200 | 37.35µs | 10.0.254.118 | GET "/healthz"
[GIN] 2022/10/12 - 21:40:12 | 200 | 26.873µs | 10.0.254.118 | GET "/healthz"
I1012 21:40:18.301545 1 request.go:665] Waited for 21.20015681s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/apisix.apache.org/v2beta3/namespaces/wshop-earum75/apisixroutes
[GIN] 2022/10/12 - 21:40:19 | 200 | 24.12µs | 10.0.254.118 | GET "/healthz"
[GIN] 2022/10/12 - 21:40:22 | 200 | 27.596µs | 10.0.254.118 | GET "/healthz"
I1012 21:40:28.301788 1 request.go:665] Waited for 31.200280131s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/apisix.apache.org/v2beta3/namespaces/wshop-leader63/apisixroutes
[GIN] 2022/10/12 - 21:40:29 | 200 | 40.844µs | 10.0.254.118 | GET "/healthz"
[GIN] 2022/10/12 - 21:40:32 | 200 | 26.052µs | 10.0.254.118 | GET "/healthz"
I1012 21:40:38.501964 1 request.go:665] Waited for 41.400382216s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/apisix.apache.org/v2beta3/namespaces/wshop-alcoholism27/apisixroutes
[GIN] 2022/10/12 - 21:40:39 | 200 | 23.659µs | 10.0.254.118 | GET "/healthz"
[GIN] 2022/10/12 - 21:40:42 | 200 | 29.537µs | 10.0.254.118 | GET "/healthz"
I1012 21:40:48.701165 1 request.go:665] Waited for 51.599499253s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/apisix.apache.org/v2beta3/namespaces/wshop-misconception18/apisixroutes
[GIN] 2022/10/12 - 21:40:49 | 200 | 24.521µs | 10.0.254.118 | GET "/healthz"
[GIN] 2022/10/12 - 21:40:52 | 200 | 24.267µs | 10.0.254.118 | GET "/healthz"
I1012 21:40:58.701926 1 request.go:665] Waited for 1m1.600208075s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/apisix.apache.org/v2beta3/namespaces/wshop-caroling45/apisixroutes
[GIN] 2022/10/12 - 21:40:59 | 200 | 24.771µs | 10.0.254.118 | GET "/healthz"
[GIN] 2022/10/12 - 21:41:02 | 200 | 39.746µs | 10.0.254.118 | GET "/healthz"
I1012 21:41:08.901096 1 request.go:665] Waited for 1m11.799295833s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/apisix.apache.org/v2beta3/namespaces/wshop-consulted24/apisixroutes
```
and i debug the code found that is the provider_init.go
https://github.com/apache/apisix-ingress-controller/blob/8a17eea26e96570dd1054f258e484bf7814627eb/pkg/providers/apisix/provider_init.go#L141
### Environment
- your apisix-ingress-controller version (output of apisix-ingress-controller version --long):
- your Kubernetes cluster version (output of kubectl version):
- if you run apisix-ingress-controller in Bare-metal environment, also show your OS version (uname -a):
Version: 1.4.1
Git SHA: no-git-module
Go Version: go1.16.15
Building OS/Arch: linux/amd64
Running OS/Arch: linux/amd64
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.6", GitCommit:"ad3338546da947756e8a88aa6822e9c11e7eac22", GitTreeState:"clean", BuildDate:"2022-04-14T08:49:13Z", GoVersion:"go1.17.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.16-eks-a05fdea", GitCommit:"a05fdea9a04815782f9c4b08eed73bb2d78b07e3", GitTreeState:"clean", BuildDate:"2022-06-09T21:50:47Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}
### Minimal test code / Steps to reproduce
1. create many namespaces
### Actual result
ingress controller stuck
### Error log
```
I1012 21:39:58.101271 1 request.go:665] Waited for 1.000058582s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/apisix.apache.org/v2beta3/namespaces/wshop-disagreement42/apisixroutes
[GIN] 2022/10/12 - 21:39:59 | 200 | 66.378µs | 10.0.254.118 | GET "/healthz"
[GIN] 2022/10/12 - 21:40:02 | 200 | 48.148µs | 10.0.254.118 | GET "/healthz"
I1012 21:40:08.101818 1 request.go:665] Waited for 11.000514374s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/apisix.apache.org/v2beta3/namespaces/wshop-negligee64/apisixroutes
[GIN] 2022/10/12 - 21:40:09 | 200 | 37.35µs | 10.0.254.118 | GET "/healthz"
[GIN] 2022/10/12 - 21:40:12 | 200 | 26.873µs | 10.0.254.118 | GET "/healthz"
I1012 21:40:18.301545 1 request.go:665] Waited for 21.20015681s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/apisix.apache.org/v2beta3/namespaces/wshop-earum75/apisixroutes
[GIN] 2022/10/12 - 21:40:19 | 200 | 24.12µs | 10.0.254.118 | GET "/healthz"
[GIN] 2022/10/12 - 21:40:22 | 200 | 27.596µs | 10.0.254.118 | GET "/healthz"
I1012 21:40:28.301788 1 request.go:665] Waited for 31.200280131s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/apisix.apache.org/v2beta3/namespaces/wshop-leader63/apisixroutes
[GIN] 2022/10/12 - 21:40:29 | 200 | 40.844µs | 10.0.254.118 | GET "/healthz"
[GIN] 2022/10/12 - 21:40:32 | 200 | 26.052µs | 10.0.254.118 | GET "/healthz"
I1012 21:40:38.501964 1 request.go:665] Waited for 41.400382216s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/apisix.apache.org/v2beta3/namespaces/wshop-alcoholism27/apisixroutes
[GIN] 2022/10/12 - 21:40:39 | 200 | 23.659µs | 10.0.254.118 | GET "/healthz"
[GIN] 2022/10/12 - 21:40:42 | 200 | 29.537µs | 10.0.254.118 | GET "/healthz"
I1012 21:40:48.701165 1 request.go:665] Waited for 51.599499253s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/apisix.apache.org/v2beta3/namespaces/wshop-misconception18/apisixroutes
[GIN] 2022/10/12 - 21:40:49 | 200 | 24.521µs | 10.0.254.118 | GET "/healthz"
[GIN] 2022/10/12 - 21:40:52 | 200 | 24.267µs | 10.0.254.118 | GET "/healthz"
I1012 21:40:58.701926 1 request.go:665] Waited for 1m1.600208075s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/apisix.apache.org/v2beta3/namespaces/wshop-caroling45/apisixroutes
[GIN] 2022/10/12 - 21:40:59 | 200 | 24.771µs | 10.0.254.118 | GET "/healthz"
[GIN] 2022/10/12 - 21:41:02 | 200 | 39.746µs | 10.0.254.118 | GET "/healthz"
I1012 21:41:08.901096 1 request.go:665] Waited for 1m11.799295833s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/apisix.apache.org/v2beta3/namespaces/wshop-consulted24/apisixroutes
```
### Expected result
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] tao12345666333 commented on issue #1378: bug: init stuck cause many namespaces
Posted by GitBox <gi...@apache.org>.
tao12345666333 commented on issue #1378:
URL: https://github.com/apache/apisix-ingress-controller/issues/1378#issuecomment-1278593456
@shareinto Are you interested in submitting a PR to fix it? Feel free to contact me if you need any help
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] tao12345666333 commented on issue #1378: bug: init stuck cause many namespaces
Posted by GitBox <gi...@apache.org>.
tao12345666333 commented on issue #1378:
URL: https://github.com/apache/apisix-ingress-controller/issues/1378#issuecomment-1278592120
yes.We can do a quick fix.
Just look at the currently watched namespaces.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] tao12345666333 closed issue #1378: bug: init stuck cause many namespaces
Posted by GitBox <gi...@apache.org>.
tao12345666333 closed issue #1378: bug: init stuck cause many namespaces
URL: https://github.com/apache/apisix-ingress-controller/issues/1378
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] AlinsRan commented on issue #1378: bug: init stuck cause many namespaces
Posted by GitBox <gi...@apache.org>.
AlinsRan commented on issue #1378:
URL: https://github.com/apache/apisix-ingress-controller/issues/1378#issuecomment-1278523782
This can be optimized by listing all the resources of the watch instead of polling all the ns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] AlinsRan commented on issue #1378: bug: init stuck cause many namespaces
Posted by GitBox <gi...@apache.org>.
AlinsRan commented on issue #1378:
URL: https://github.com/apache/apisix-ingress-controller/issues/1378#issuecomment-1279666713
We can use the local cache to do this, that is, client-go Informer
1. Make sure the Informer is started.
2. Wait for API Server synchronize data to local cache.
3. To be able to get or compare these objects from informer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] AlinsRan commented on issue #1378: bug: init stuck cause many namespaces
Posted by GitBox <gi...@apache.org>.
AlinsRan commented on issue #1378:
URL: https://github.com/apache/apisix-ingress-controller/issues/1378#issuecomment-1342411158
Hi @shareinto !
I sent you an email.
I still have some confuse about these issues. Would you do me a favor? I want to know more information. Looking forward to your reply.
Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org