You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2022/10/18 10:06:36 UTC
[GitHub] [apisix-ingress-controller] shareinto opened a new issue, #1393: bug: ingress tls secret not found
shareinto opened a new issue, #1393:
URL: https://github.com/apache/apisix-ingress-controller/issues/1393
### Issue description
When the tls of the ingress is translated, it needs to obtain the secret from the lister, but the informer may not be completed synchronously at this time, so the acquisition of the secret fails.
What I can be sure of is that the secret does exist
```
k -n wshop-cocked92 get secret wshop-wildcard ──(Tue,Oct18)─┘
NAME TYPE DATA AGE
wshop-wildcard kubernetes.io/tls 2 454d
```
error log
```
2022-10-18T05:50:54-04:00 error translation/translator.go:384 failed to translate ingress tls to apisix tls {"error": "secret \"mshop-wildcard\" not found", "ingress": "&Ingress{ObjectMeta:{inner-ingress wshop-cocked92 /apis/extensions/v1beta1/namespaces/wshop-cocked92/ingresses/inner-ingress 3e06dc73-586c-48df-a28a-57ad8ca37727 546777417 6 2021-07-20 10:54:13 -0400 EDT ..."}
```
https://github.com/apache/apisix-ingress-controller/blob/5f2c39815f483b30f4ae3305556564244563a589/pkg/providers/apisix/translation/apisix_ssl.go#L70
### Environment
- your apisix-ingress-controller version (output of apisix-ingress-controller version --long):
Version: 1.5.0
Git SHA: https://github.com/apache/apisix-ingress-controller/commit/f23454fa30e92a4e4392a8c5b2b0ffd2d3bf80b4
Go Version: go1.19.2
Building OS/Arch: linux/amd64
Running OS/Arch: linux/amd64
- your Kubernetes cluster version (output of kubectl version):
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.6", GitCommit:"ad3338546da947756e8a88aa6822e9c11e7eac22", GitTreeState:"clean", BuildDate:"2022-04-14T08:49:13Z", GoVersion:"go1.17.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.16-eks-a05fdea", GitCommit:"a05fdea9a04815782f9c4b08eed73bb2d78b07e3", GitTreeState:"clean", BuildDate:"2022-06-09T21:50:47Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.23) and server (1.19) exceeds the supported minor version skew of +/-1
- if you run apisix-ingress-controller in Bare-metal environment, also show your OS version (uname -a):
### Minimal test code / Steps to reproduce
1. create a lot of secrets
2. create a ingress with tls
### Actual result
ingress translate failed
### Error log
```
2022-10-18T05:50:54-04:00 error translation/translator.go:384 failed to translate ingress tls to apisix tls {"error": "secret \"mshop-wildcard\" not found", "ingress": "&Ingress{ObjectMeta:{inner-ingress wshop-cocked92 /apis/extensions/v1beta1/namespaces/wshop-cocked92/ingresses/inner-ingress 3e06dc73-586c-48df-a28a-57ad8ca37727 546777417 6 2021-07-20 10:54:13 -0400 EDT <nil> <nil> map[] map[field.cattle.io/publicEndpoints:[{\"addresses\":[\"\"],\"port\":443,\"protocol\":\"HTTPS\",\"serviceName\":\"wshop-cocked92:mshop-svc\",\"ingressName\":\"wshop-cocked92:inner-ingress\",\"hostname\":\"cocked92.hotishop.com\",\"path\":\"/\",\"allNodes\":false}] kubernetes.io/ingress.class:nginx nginx.ingress.kubernetes.io/proxy-body-size:20M nginx.ingress.kubernetes.io/ssl-redirect:true] [] [] []},Spec:IngressSpec{Backend:nil,TLS:[]IngressTLS{IngressTLS{Hosts:[cocked92.hotishop.com],SecretName:mshop-wildcard,},},Rules:[]IngressRule{IngressRule{Host:cocked92.hotishop.com,IngressRuleV
alue:IngressRuleValue{HTTP:&HTTPIngressRuleValue{Paths:[]HTTPIngressPath{HTTPIngressPath{Path:/,Backend:IngressBackend{ServiceName:mshop-svc,ServicePort:{0 80 },Resource:nil,},PathType:*ImplementationSpecific,},},},},},},IngressClassName:nil,},Status:IngressStatus{LoadBalancer:{[{ a848a324cb9474bfbbd0c32d417d4de3-562776765955e017.elb.us-east-2.amazonaws.com []}]},},}"}
```
### Expected result
ingress translate successfully
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] tao12345666333 commented on issue #1393: bug: ingress tls secret not found
Posted by GitBox <gi...@apache.org>.
tao12345666333 commented on issue #1393:
URL: https://github.com/apache/apisix-ingress-controller/issues/1393#issuecomment-1290240058
This problem is worth fixing.
I would like to know more information. Do you have a large cluster? Or are you doing a stress test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] shareinto commented on issue #1393: perf: When a large number of resources exist, the data in the cache is not synchronized
Posted by GitBox <gi...@apache.org>.
shareinto commented on issue #1393:
URL: https://github.com/apache/apisix-ingress-controller/issues/1393#issuecomment-1336983635
Hi @AlinsRan
Do endpoints and services have the same problems as secrets?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] shareinto commented on issue #1393: bug: ingress tls secret not found
Posted by GitBox <gi...@apache.org>.
shareinto commented on issue #1393:
URL: https://github.com/apache/apisix-ingress-controller/issues/1393#issuecomment-1284798408
```
k get secrets -A | wc -l ──(Wed,Oct19)─┘
79885
```
@tao12345666333
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] AlinsRan commented on issue #1393: perf: When a large number of resources exist, the data in the cache is not synchronized
Posted by GitBox <gi...@apache.org>.
AlinsRan commented on issue #1393:
URL: https://github.com/apache/apisix-ingress-controller/issues/1393#issuecomment-1337021120
I think the informer should be started uniformly and wait for synchronization. Instead of maintaining dependencies on these resources in many places. Assuming a long dependency chain, I don't think any of them will perform very differently between them.
However, it should be noted that when the informer starts, it needs to ensure that the event handler has been registered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] shareinto commented on issue #1393: perf: When a large number of resources exist, the data in the cache is not synchronized
Posted by GitBox <gi...@apache.org>.
shareinto commented on issue #1393:
URL: https://github.com/apache/apisix-ingress-controller/issues/1393#issuecomment-1337081651
> > Hi @shareinto. I think if you don't want to make big changes, you just need to add a waiting condition here https://github.com/apache/apisix-ingress-controller/blob/master/pkg/providers/ingress/ingress.go#L90 add `secretInformer.HasSynced`.
> > What do you think?
>
> Yes, I agree with your method, but there is a problem that it is difficult to get secretInformer in the ingress provider
I found that secretInformer and so on are initialized in the global scope, so I can pass it to the Ingress Provider, I made a new commit, you can see if it workshttps://github.com/apache/apisix-ingress-controller/pull/1394/commits/df91ea69b10e683af13068bc509559e2659c3ce8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] AlinsRan commented on issue #1393: perf: When a large number of resources exist, the data in the cache is not synchronized
Posted by GitBox <gi...@apache.org>.
AlinsRan commented on issue #1393:
URL: https://github.com/apache/apisix-ingress-controller/issues/1393#issuecomment-1338693409
how to add code here https://github.com/apache/apisix-ingress-controller/blob/master/pkg/providers/controller.go#L388 ?
```golang
c.kubeFactory.Start(ctx.Done())
c.apisixFactory.Start(ctx.Done())
c.kubeFactory.WaitForCacheSync(ctx.Done())
c.apisixFactory.WaitForCacheSync(ctx.Done())
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] shareinto commented on issue #1393: perf: When a large number of resources exist, the data in the cache is not synchronized
Posted by GitBox <gi...@apache.org>.
shareinto commented on issue #1393:
URL: https://github.com/apache/apisix-ingress-controller/issues/1393#issuecomment-1291533279
> This problem is worth fixing.
>
> I would like to know more information. Do you have a large cluster? Or are you doing a stress test?
yes i have a production cluster ,There are about 9000 ingresses, 80000 secrets and 15000 namespaces。I did remote debugging on this cluster. The exception was indeed that when the lister obtained the secret, a secret not found exception occurred. After I added waitforsynced to the informer, the exception disappeared.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] tao12345666333 closed issue #1393: perf: When a large number of resources exist, the data in the cache is not synchronized
Posted by GitBox <gi...@apache.org>.
tao12345666333 closed issue #1393: perf: When a large number of resources exist, the data in the cache is not synchronized
URL: https://github.com/apache/apisix-ingress-controller/issues/1393
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] tao12345666333 commented on issue #1393: bug: ingress tls secret not found
Posted by GitBox <gi...@apache.org>.
tao12345666333 commented on issue #1393:
URL: https://github.com/apache/apisix-ingress-controller/issues/1393#issuecomment-1284795651
> create a lot of secrets
Thanks for your report. Can you describe it more clearly? hundreds of? Thousands?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] shareinto commented on issue #1393: perf: When a large number of resources exist, the data in the cache is not synchronized
Posted by GitBox <gi...@apache.org>.
shareinto commented on issue #1393:
URL: https://github.com/apache/apisix-ingress-controller/issues/1393#issuecomment-1337021975
> Hi @shareinto. I think if you don't want to make big changes, you just need to add a waiting condition here https://github.com/apache/apisix-ingress-controller/blob/master/pkg/providers/ingress/ingress.go#L90 add `secretInformer.HasSynced`.
>
> What do you think?
Yes, I agree with your method, but there is a problem that it is difficult to get secretInformer in the ingress provider
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] AlinsRan commented on issue #1393: perf: When a large number of resources exist, the data in the cache is not synchronized
Posted by GitBox <gi...@apache.org>.
AlinsRan commented on issue #1393:
URL: https://github.com/apache/apisix-ingress-controller/issues/1393#issuecomment-1336967425
I think if you don't want to make big changes, you just need to add a waiting condition here
https://github.com/apache/apisix-ingress-controller/blob/master/pkg/providers/ingress/ingress.go#L90
add `secretInformer.HasSynced`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] AlinsRan commented on issue #1393: perf: When a large number of resources exist, the data in the cache is not synchronized
Posted by GitBox <gi...@apache.org>.
AlinsRan commented on issue #1393:
URL: https://github.com/apache/apisix-ingress-controller/issues/1393#issuecomment-1337012815
> Hi @AlinsRan Do endpoints and services have the same problems as secrets?
I think there is this problem and this module needs to be refactored.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] AlinsRan commented on issue #1393: perf: When a large number of resources exist, the data in the cache is not synchronized
Posted by GitBox <gi...@apache.org>.
AlinsRan commented on issue #1393:
URL: https://github.com/apache/apisix-ingress-controller/issues/1393#issuecomment-1337070765
> I think the informers should be started uniformly and wait for synchronization. Instead of maintaining dependencies on these resources in many places.
>
> However, it should be noted that when the informer starts, it needs to ensure that the event handle function has been registered. Otherwise, this would cause some events to be lost.
>
> 
#L388 Maybe you can start here and wait for synchronization. https://github.com/apache/apisix-ingress-controller/blob/master/pkg/providers/controller.go
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org