You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2021/05/13 10:32:35 UTC
[GitHub] [apisix-ingress-controller] slene opened a new issue #449: bug: When controller become to leader from candidate code panic error
slene opened a new issue #449:
URL: https://github.com/apache/apisix-ingress-controller/issues/449
### Issue description
### Environment
* apisix-ingress-controller, latest master branch
* k8s v1.20.2
* apisix 2.5
### Minimal test code / Steps to reproduce the issue
1. start apisix-ingress first
2. stop kube-apiserver
3. controller now is running as a candidate
4. start kube-apiserver
5. controller now is running as leader
6. panic: "close of closed channel" (close of closed channel)
### What's the actual result? (including assertion message & call stack if applicable)
```
E0513 17:38:10.666973 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.2/tools/cache/reflector.go:167: Failed to watch *v2alpha1.ApisixClusterConfig: failed to list *v2alpha1.ApisixClusterConfig: the server could not find the requested resource (get apisixclusterconfigs.apisix.apache.org)
W0513 17:38:18.852172 1 reflector.go:436] pkg/mod/k8s.io/client-go@v0.20.2/tools/cache/reflector.go:167: watch of *v1.Secret ended with: very short watch: pkg/mod/k8s.io/client-go@v0.20.2/tools/cache/reflector.go:167: Unexpected watch close - watch lasted less than a second and no items received
E0513 17:38:19.692055 1 leaderelection.go:325] error retrieving resource lock default/ingress-apisix-leader: Get "https://172.24.78.235:8443/apis/coordination.k8s.io/v1/namespaces/default/leases/ingress-apisix-leader": dial tcp 172.24.78.235:8443: connect: connection refused
E0513 17:38:20.598540 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.2/tools/cache/reflector.go:167: Failed to watch *v1.Secret: failed to list *v1.Secret: Get "https://172.24.78.235:8443/api/v1/secrets?resourceVersion=354595": dial tcp 172.24.78.235:8443: connect: connection refused
E0513 17:38:21.692777 1 leaderelection.go:325] error retrieving resource lock default/ingress-apisix-leader: Get "https://172.24.78.235:8443/apis/coordination.k8s.io/v1/namespaces/default/leases/ingress-apisix-leader": dial tcp 172.24.78.235:8443: connect: connection refused
E0513 17:38:23.692718 1 leaderelection.go:325] error retrieving resource lock default/ingress-apisix-leader: Get "https://172.24.78.235:8443/apis/coordination.k8s.io/v1/namespaces/default/leases/ingress-apisix-leader": dial tcp 172.24.78.235:8443: connect: connection refused
2021-05-13T17:38:24+08:00 info ingress/controller.go:246 LeaderElection {"message": " stopped leading", "event_type": "Normal"}
I0513 17:38:24.691934 1 leaderelection.go:278] failed to renew lease default/ingress-apisix-leader: timed out waiting for the condition
E0513 17:38:24.692023 1 leaderelection.go:301] Failed to release lock: resource name may not be empty
2021-05-13T17:38:24+08:00 info ingress/controller.go:293 controller now is running as a candidate {"namespace": "default", "pod": ""}
I0513 17:38:24.692080 1 leaderelection.go:243] attempting to acquire leader lease default/ingress-apisix-leader...
2021-05-13T17:38:24+08:00 info ingress/apisix_tls.go:70 ApisixTls controller exited
2021-05-13T17:38:24+08:00 info ingress/apisix_route.go:70 ApisixRoute controller exited
2021-05-13T17:38:24+08:00 info ingress/endpoint.go:86 endpoints controller exited
2021-05-13T17:38:24+08:00 info ingress/apisix_upstream.go:70 ApisixUpstream controller exited
2021-05-13T17:38:24+08:00 info ingress/secret.go:74 secret controller exited
2021-05-13T17:38:24+08:00 info ingress/ingress.go:71 ingress controller exited
2021-05-13T17:38:24+08:00 error ingress/apisix_cluster_config.go:58 cache sync failed
2021-05-13T17:38:24+08:00 info ingress/apisix_cluster_config.go:59 ApisixClusterConfig controller exited
E0513 17:38:24.692344 1 leaderelection.go:325] error retrieving resource lock default/ingress-apisix-leader: Get "https://172.24.78.235:8443/apis/coordination.k8s.io/v1/namespaces/default/leases/ingress-apisix-leader": dial tcp 172.24.78.235:8443: connect: connection refused
E0513 17:38:29.059509 1 leaderelection.go:325] error retrieving resource lock default/ingress-apisix-leader: Get "https://172.24.78.235:8443/apis/coordination.k8s.io/v1/namespaces/default/leases/ingress-apisix-leader": dial tcp 172.24.78.235:8443: connect: connection refused
E0513 17:38:31.265768 1 leaderelection.go:325] error retrieving resource lock default/ingress-apisix-leader: Get "https://172.24.78.235:8443/apis/coordination.k8s.io/v1/namespaces/default/leases/ingress-apisix-leader": dial tcp 172.24.78.235:8443: connect: connection refused
E0513 17:38:34.873517 1 leaderelection.go:325] error retrieving resource lock default/ingress-apisix-leader: Get "https://172.24.78.235:8443/apis/coordination.k8s.io/v1/namespaces/default/leases/ingress-apisix-leader": dial tcp 172.24.78.235:8443: connect: connection refused
E0513 17:38:37.442765 1 leaderelection.go:325] error retrieving resource lock default/ingress-apisix-leader: Get "https://172.24.78.235:8443/apis/coordination.k8s.io/v1/namespaces/default/leases/ingress-apisix-leader": dial tcp 172.24.78.235:8443: connect: connection refused
E0513 17:38:40.728304 1 leaderelection.go:325] error retrieving resource lock default/ingress-apisix-leader: Get "https://172.24.78.235:8443/apis/coordination.k8s.io/v1/namespaces/default/leases/ingress-apisix-leader": dial tcp 172.24.78.235:8443: connect: connection refused
E0513 17:38:43.178480 1 leaderelection.go:325] error retrieving resource lock default/ingress-apisix-leader: Get "https://172.24.78.235:8443/apis/coordination.k8s.io/v1/namespaces/default/leases/ingress-apisix-leader": dial tcp 172.24.78.235:8443: connect: connection refused
2021-05-13T17:38:47+08:00 info ingress/controller.go:246 LeaderElection {"message": " became leader", "event_type": "Normal"}
I0513 17:38:47.031012 1 leaderelection.go:253] successfully acquired lease default/ingress-apisix-leader
2021-05-13T17:38:47+08:00 info ingress/controller.go:321 controller now is running as leader {"namespace": "default", "pod": ""}
2021-05-13T17:38:47+08:00 info ingress/secret.go:60 secret controller started
2021-05-13T17:38:47+08:00 info ingress/endpoint.go:59 endpoints controller started
2021-05-13T17:38:47+08:00 info ingress/apisix_upstream.go:58 ApisixUpstream controller started
2021-05-13T17:38:47+08:00 info ingress/ingress.go:59 ingress controller started
2021-05-13T17:38:47+08:00 info ingress/apisix_route.go:57 ApisixRoute controller started
2021-05-13T17:38:47+08:00 info ingress/apisix_cluster_config.go:55 ApisixClusterConfig controller started
2021-05-13T17:38:47+08:00 info ingress/apisix_tls.go:58 ApisixTls controller started
E0513 17:38:47.032363 1 runtime.go:78] Observed a panic: "close of closed channel" (close of closed channel)
goroutine 678 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x16f8a20, 0x1b85f10)
C:/Users/slene/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/runtime/runtime.go:74 +0x92
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
C:/Users/slene/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/runtime/runtime.go:48 +0x82
panic(0x16f8a20, 0x1b85f10)
c:/go/src/runtime/panic.go:969 +0x166
k8s.io/client-go/tools/cache.(*processorListener).pop(0xc0002f0e00)
C:/Users/slene/go/pkg/mod/k8s.io/client-go@v0.20.2/tools/cache/shared_informer.go:752 +0x2ba
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc0001f8f40, 0xc00010d4d0)
C:/Users/slene/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:73 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
C:/Users/slene/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:71 +0x62
panic: close of closed channel [recovered]
panic: close of closed channel
goroutine 678 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
C:/Users/slene/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/runtime/runtime.go:55 +0x105
panic(0x16f8a20, 0x1b85f10)
c:/go/src/runtime/panic.go:969 +0x166
k8s.io/client-go/tools/cache.(*processorListener).pop(0xc0002f0e00)
C:/Users/slene/go/pkg/mod/k8s.io/client-go@v0.20.2/tools/cache/shared_informer.go:752 +0x2ba
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc0001f8f40, 0xc00010d4d0)
C:/Users/slene/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:73 +0x51
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
C:/Users/slene/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:71 +0x62
```
### What's the expected result?
https://github.com/apache/apisix-ingress-controller/blob/f199cdb5f5bfe3cb5acb19dc1903b1f5f426a353/pkg/ingress/controller.go#L190-L205
https://github.com/apache/apisix-ingress-controller/blob/f199cdb5f5bfe3cb5acb19dc1903b1f5f426a353/pkg/ingress/controller.go#L344-L367
When controller become to leader from candidate. That code use SharedIndexInformer not recycled. And reused in Controller.run. But context already canceled. So code panic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] tokers commented on issue #449: bug: When controller become to leader from candidate code panic error
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #449:
URL: https://github.com/apache/apisix-ingress-controller/issues/449#issuecomment-840491646
@slene Thanks for your report and analysis, would you like to submit a PR to fix it? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] gxthrj closed issue #449: bug: When controller become to leader from candidate code panic error
Posted by GitBox <gi...@apache.org>.
gxthrj closed issue #449:
URL: https://github.com/apache/apisix-ingress-controller/issues/449
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org