You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2021/05/13 09:29:51 UTC
[GitHub] [apisix-ingress-controller] slene opened a new issue #448: bug: Always sync ingress failed if apisix cluster not ready when apisix-ingress start.
slene opened a new issue #448:
URL: https://github.com/apache/apisix-ingress-controller/issues/448
### Issue description
### Environment
* apisix-ingress-controller, latest master branch
* k8s v1.20.2
* apisix 2.5
### Minimal test code / Steps to reproduce the issue
1. start apisix-ingress first
2. then start apisix
3. change some ingress resources
4. always sync ingress failed
### What's the actual result? (including assertion message & call stack if applicable)
```
I0513 17:07:52.216990 1 leaderelection.go:243] attempting to acquire leader lease default/ingress-apisix-leader...
2021-05-13T17:07:52+08:00 info ingress/controller.go:246 LeaderElection {"message": " became leader", "event_type": "Normal"}
I0513 17:07:52.271697 1 leaderelection.go:253] successfully acquired lease default/ingress-apisix-leader
2021-05-13T17:07:52+08:00 info ingress/controller.go:321 controller now is running as leader {"namespace": "default", "pod": ""}
2021-05-13T17:07:52+08:00 warn apisix/cluster.go:252 waiting cluster default to ready, it may takes a while
2021-05-13T17:07:52+08:00 info apisix/cluster.go:115 syncing cache {"cluster": "default"}
2021-05-13T17:07:52+08:00 debug apisix/route.go:111 try to list routes in APISIX {"cluster": "default", "url": "http://127.0.0.1:9182/apisix/admin/routes"}
2021-05-13T17:07:52+08:00 error apisix/route.go:117 failed to list routes: Get "http://127.0.0.1:9182/apisix/admin/routes": dial tcp 127.0.0.1:9182: connect: connection refused
2021-05-13T17:07:52+08:00 error apisix/cluster.go:159 failed to list route in APISIX: Get "http://127.0.0.1:9182/apisix/admin/routes": dial tcp 127.0.0.1:9182: connect: connection refused
2021-05-13T17:07:52+08:00 error apisix/cluster.go:127 failed to sync cache {"cost_time": "6.631413ms", "cluster": "default"}
2021-05-13T17:07:52+08:00 warn apisix/cluster.go:258 cluster default now is ready, cost time 6.661213ms
2021-05-13T17:07:52+08:00 info ingress/secret.go:60 secret controller started
2021-05-13T17:07:52+08:00 info ingress/endpoint.go:59 endpoints controller started
2021-05-13T17:07:52+08:00 info ingress/apisix_tls.go:58 ApisixTls controller started
2021-05-13T17:07:52+08:00 info ingress/apisix_upstream.go:58 ApisixUpstream controller started
2021-05-13T17:07:52+08:00 info ingress/ingress.go:59 ingress controller started
2021-05-13T17:07:52+08:00 info ingress/apisix_route.go:57 ApisixRoute controller started
...
2021-05-13T17:09:57+08:00 error ingress/ingress.go:176 failed to sync ingress artifacts {"error": "2 errors occurred:\n\t* Get \"http://127.0.0.1:9182/apisix/admin/routes\": dial tcp 127.0.0.1:9182: connect: connection refused\n\t* Get \"http://127.0.0.1:9182/apisix/admin/routes\": dial tcp 127.0.0.1:9182: connect: connection refused\n\n"}
...
// start apisix
// change some ingress resources
// like kubectl create ing httpbinxx --rule="/*=httpbin:80" --class=apisix
...
2021-05-13T17:11:19+08:00 debug ingress/ingress.go:138 translated ingress resource to a couple of routes and upstreams {"ingress": {}, "routes": [{"id":"95d36b91","name":"ingress__/","uris":["/","/*"],"upstream_id":"5ce57b8e"}], "upstreams": [{"id":"5ce57b8e","name":"default_httpbin_80","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","labels":{"managed-by":"apisix-ingress-controller"},"type":"roundrobin","nodes":[{"host":"10.2.0.4","port":80,"weight":100}],"scheme":"http"}]}
2021-05-13T17:11:19+08:00 debug apisix/route.go:177 try to delete route {"id": "95d36b91", "name": "ingress__/", "cluster": "default", "url": "http://127.0.0.1:9182/apisix/admin/routes"}
2021-05-13T17:11:19+08:00 debug apisix/upstream.go:169 try to delete upstream {"id": "5ce57b8e", "name": "default_httpbin_80", "cluster": "default", "url": "http://127.0.0.1:9182/apisix/admin/upstreams"}
2021-05-13T17:11:19+08:00 error ingress/ingress.go:176 failed to sync ingress artifacts {"error": "2 errors occurred:\n\t* Get \"http://127.0.0.1:9182/apisix/admin/routes\": dial tcp 127.0.0.1:9182: connect: connection refused\n\t* Get \"http://127.0.0.1:9182/apisix/admin/routes\": dial tcp 127.0.0.1:9182: connect: connection refused\n\n"}
2021-05-13T17:11:19+08:00 warn ingress/ingress.go:189 sync ingress failed, will retry {"object": {"Type":3,"Object":{"Key":"default/httpbinxx","GroupVersion":"extensions/v1beta1","OldObject":null},"Tombstone":{}}, "error": "2 errors occurred:\n\t* Get \"http://127.0.0.1:9182/apisix/admin/routes\": dial tcp 127.0.0.1:9182: connect: connection refused\n\t* Get \"http://127.0.0.1:9182/apisix/admin/routes\": dial tcp 127.0.0.1:9182: connect: connection refused\n\n"}
```
### What's the expected result?
https://github.com/apache/apisix-ingress-controller/blob/f613f36bab842089ab54f681ed1b4bf126e4c162/pkg/apisix/cluster.go#L133-L142
https://github.com/apache/apisix-ingress-controller/blob/f613f36bab842089ab54f681ed1b4bf126e4c162/pkg/apisix/cluster.go#L242-L244
https://github.com/apache/apisix-ingress-controller/blob/f613f36bab842089ab54f681ed1b4bf126e4c162/pkg/apisix/route.go#L141-L151
https://github.com/apache/apisix-ingress-controller/pull/176 say.
> Also, controller will try to sync (with an exponential back off strategy) to the APISIX cluster when the cluster was just added. All operations will fail if the cluster is not ready (the initial sync is failure), we'll add a recover mechanism once the APISIX cluster recovered, the controller shall detect it.
I have not found any recover code if cluster is not ready. ExponentialBackoff will return if syncCacheOnce return an error. Can it be use PollImmediateInfinite instead ? It can wait for success and return. If possible, I will submit the code.
```
err := wait.PollImmediateInfinite(time.Second*3, func() (bool, error) {
// wait until success
if done, _ := c.syncCacheOnce(); done {
return true, nil
}
return false, nil
})
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] gxthrj closed issue #448: bug: Always sync ingress failed if apisix cluster not ready when apisix-ingress start.
Posted by GitBox <gi...@apache.org>.
gxthrj closed issue #448:
URL: https://github.com/apache/apisix-ingress-controller/issues/448
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] tokers commented on issue #448: bug: Always sync ingress failed if apisix cluster not ready when apisix-ingress start.
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #448:
URL: https://github.com/apache/apisix-ingress-controller/issues/448#issuecomment-840518933
I have submitted a PR to fix this: https://github.com/apache/apisix-ingress-controller/pull/450.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] gxthrj commented on issue #448: bug: Always sync ingress failed if apisix cluster not ready when apisix-ingress start.
Posted by GitBox <gi...@apache.org>.
gxthrj commented on issue #448:
URL: https://github.com/apache/apisix-ingress-controller/issues/448#issuecomment-840451975
Yes, It is a bug, when connecting refused, need to retry.
@tokers PTAL, If you have time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] tokers commented on issue #448: bug: Always sync ingress failed if apisix cluster not ready when apisix-ingress start.
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #448:
URL: https://github.com/apache/apisix-ingress-controller/issues/448#issuecomment-840501067
@slene Thanks for your report.
I have captured this bug, after we got a signal from `c.cacheSynced` channel, we should in turn check the `c.cacheSyncErr` as the caching synchronization is asynchronous.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] gxthrj closed issue #448: bug: Always sync ingress failed if apisix cluster not ready when apisix-ingress start.
Posted by GitBox <gi...@apache.org>.
gxthrj closed issue #448:
URL: https://github.com/apache/apisix-ingress-controller/issues/448
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org