You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by "mrmm (via GitHub)" <gi...@apache.org> on 2023/08/03 15:08:21 UTC
[GitHub] [apisix-ingress-controller] mrmm opened a new issue, #1914: bug: Ingress controller crash loop when using ingress resources after restart
mrmm opened a new issue, #1914:
URL: https://github.com/apache/apisix-ingress-controller/issues/1914
### Current Behavior
When restarting the pod running the ingress controller it crashes randomly with the error.
### Expected Behavior
Starts correctly and starts watching the ingress resources.
### Error Logs
```
apisix-ingress-controller-545d9f8cdd-h69zg apisix-ingress-controller 2023-08-03T09:07:23Z error ingress/ingress.go:581 cache recorded invalid ingress version key {"key": "default/ingress-v1-test"}
apisix-ingress-controller-545d9f8cdd-h69zg apisix-ingress-controller panic: runtime error: index out of range [1] with length 1
apisix-ingress-controller-545d9f8cdd-h69zg apisix-ingress-controller
apisix-ingress-controller-545d9f8cdd-h69zg apisix-ingress-controller goroutine 353 [running]:
apisix-ingress-controller-545d9f8cdd-h69zg apisix-ingress-controller github.com/apache/apisix-ingress-controller/pkg/providers/ingress.(*ingressController).syncSSLs(0xc000a777d0, {0x1f455d0?, 0xc000654940}, 0x1, 0xc001290080, {0xc001358140, 0x1f}, {0xc0008c7f20, 0x18}, 0xc0000b77a0)
apisix-ingress-controller-545d9f8cdd-h69zg apisix-ingress-controller /bitnami/blacksmith-sandox/apisix-ingress-controller-1.6.1/src/github.com/apache/apisix-ingress-controller/pkg/providers/ingress/ingress.go:586 +0xbec
apisix-ingress-controller-545d9f8cdd-h69zg apisix-ingress-controller github.com/apache/apisix-ingress-controller/pkg/providers/ingress.(*ingressController).SyncSecretChange.func1({0x1906220?, 0xc001005850?}, {0x19f03e0?, 0xc0000b77a0?})
apisix-ingress-controller-545d9f8cdd-h69zg apisix-ingress-controller /bitnami/blacksmith-sandox/apisix-ingress-controller-1.6.1/src/github.com/apache/apisix-ingress-controller/pkg/providers/ingress/ingress.go:574 +0x86
apisix-ingress-controller-545d9f8cdd-h69zg apisix-ingress-controller sync.(*Map).Range(0xc000a6e8a0?, 0xc0014b5c78)
apisix-ingress-controller-545d9f8cdd-h69zg apisix-ingress-controller /usr/local/go-1.19.11/src/sync/map.go:354 +0x2b8
apisix-ingress-controller-545d9f8cdd-h69zg apisix-ingress-controller github.com/apache/apisix-ingress-controller/pkg/providers/ingress.(*ingressController).SyncSecretChange(0xc000a777d0, {0x1f455d0, 0xc000654940}, 0xc000844340, 0xc001290080, {0xc001358140, 0x1f})
apisix-ingress-controller-545d9f8cdd-h69zg apisix-ingress-controller /bitnami/blacksmith-sandox/apisix-ingress-controller-1.6.1/src/github.com/apache/apisix-ingress-controller/pkg/providers/ingress/ingress.go:571 +0x112
apisix-ingress-controller-545d9f8cdd-h69zg apisix-ingress-controller github.com/apache/apisix-ingress-controller/pkg/providers/ingress.(*ingressProvider).SyncSecretChange(0xc001014040?, {0x1f455d0?, 0xc000654940?}, 0x1c5ced1?, 0xc00134e000?, {0xc001358140?, 0x15f6922?})
apisix-ingress-controller-545d9f8cdd-h69zg apisix-ingress-controller /bitnami/blacksmith-sandox/apisix-ingress-controller-1.6.1/src/github.com/apache/apisix-ingress-controller/pkg/providers/ingress/provider.go:100 +0x2c
apisix-ingress-controller-545d9f8cdd-h69zg apisix-ingress-controller github.com/apache/apisix-ingress-controller/pkg/providers/k8s.(*secretController).sync(0xc000263110, {0x1f455d0, 0xc000654940}, 0xc000844340)
apisix-ingress-controller-545d9f8cdd-h69zg apisix-ingress-controller /bitnami/blacksmith-sandox/apisix-ingress-controller-1.6.1/src/github.com/apache/apisix-ingress-controller/pkg/providers/k8s/secret.go:149 +0x423
apisix-ingress-controller-545d9f8cdd-h69zg apisix-ingress-controller github.com/apache/apisix-ingress-controller/pkg/providers/k8s.(*secretController).runWorker(0xc000263110, {0x1f455d0, 0xc000654940})
apisix-ingress-controller-545d9f8cdd-h69zg apisix-ingress-controller /bitnami/blacksmith-sandox/apisix-ingress-controller-1.6.1/src/github.com/apache/apisix-ingress-controller/pkg/providers/k8s/secret.go:101 +0x4a
apisix-ingress-controller-545d9f8cdd-h69zg apisix-ingress-controller created by github.com/apache/apisix-ingress-controller/pkg/providers/k8s.(*secretController).run
apisix-ingress-controller-545d9f8cdd-h69zg apisix-ingress-controller /bitnami/blacksmith-sandox/apisix-ingress-controller-1.6.1/src/github.com/apache/apisix-ingress-controller/pkg/providers/k8s/secret.go:89 +0x25e
- apisix-ingress-controller-545d9f8cdd-h69zg › apisix-ingress-controller
```
### Steps to Reproduce
1- Deploy the ingress controller with version `1.6.1`
2- Create a new ingress I have reproduced the problem with the ingress below
```
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
namespace: default
annotations:
kubernetes.io/ingress.class: apisix
cert-manager.io/cluster-issuer: letsencrypt-staging
name: ingress-v1-test
spec:
rules:
- host: test-ingress.domain.com
http:
paths:
- path: /
pathType: ImplementationSpecific
backend:
service:
name: echoserver
port:
number: 80
tls:
- hosts:
- test-ingress.domain.com
secretName: test-ingress-cert
```
3- Remove the ingress controller pod or do something to recreate the ingress controller pod
### Environment
- APISIX Ingress controller version (run `apisix-ingress-controller version --long`)
```
apisix-ingress-controller@bitnami:/$ apisix-ingress-controller version --long
Version: 1.6.1
Git SHA: f9aba39
Go Version: go1.19.11
Building OS/Arch: linux/aarch64
Running OS/Arch: linux/arm64
```
- Kubernetes cluster version (run `kubectl version`)
```
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.4", GitCommit:"fa3d7990104d7c1f16943a67f11b154b71f6a132", GitTreeState:"clean", BuildDate:"2023-07-19T12:14:48Z", GoVersion:"go1.20.6", Compiler:"gc
", Platform:"darwin/arm64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.3", GitCommit:"f18584a06fc476806da2e340e2eed960659871e8", GitTreeState:"clean", BuildDate:"2023-06-12T18:45:20Z", GoVersion:"go1.19.7", Compiler:"gc
", Platform:"linux/amd64"}
```
- OS version if running APISIX Ingress controller in a bare-metal environment (run `uname -a`)
```
Linux bitnami 5.15.49-linuxkit-pr #1 SMP PREEMPT Thu May 25 07:27:39 UTC 2023 aarch64 GNU/Linux
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] tao12345666333 commented on issue #1914: bug: Ingress controller crash loop when using ingress resources after restart
Posted by "tao12345666333 (via GitHub)" <gi...@apache.org>.
tao12345666333 commented on issue #1914:
URL: https://github.com/apache/apisix-ingress-controller/issues/1914#issuecomment-1664830665
The current version of APISIX v1.7 is nearing its end and will be released soon. Please use v1.6.0 for now or wait for the release of v1.7.0. Thanks
https://github.com/apache/apisix-ingress-controller/milestone/16
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] mrmm commented on issue #1914: bug: Ingress controller crash loop when using ingress resources after restart
Posted by "mrmm (via GitHub)" <gi...@apache.org>.
mrmm commented on issue #1914:
URL: https://github.com/apache/apisix-ingress-controller/issues/1914#issuecomment-1678148330
Closing, for whoever encounters this issue I am using a custom-built image with the HEAD main branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] mrmm closed issue #1914: bug: Ingress controller crash loop when using ingress resources after restart
Posted by "mrmm (via GitHub)" <gi...@apache.org>.
mrmm closed issue #1914: bug: Ingress controller crash loop when using ingress resources after restart
URL: https://github.com/apache/apisix-ingress-controller/issues/1914
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] mrmm commented on issue #1914: bug: Ingress controller crash loop when using ingress resources after restart
Posted by "mrmm (via GitHub)" <gi...@apache.org>.
mrmm commented on issue #1914:
URL: https://github.com/apache/apisix-ingress-controller/issues/1914#issuecomment-1667443594
@tao12345666333 Thanks for the response 🙏
Can you please link the known issue in the release note so the other users of the project can know?
PS: My workaround is to build the HEAD version of the `apisix-ingress-controller` and use it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] mrmm commented on issue #1914: bug: Ingress controller crash loop when using ingress resources after restart
Posted by "mrmm (via GitHub)" <gi...@apache.org>.
mrmm commented on issue #1914:
URL: https://github.com/apache/apisix-ingress-controller/issues/1914#issuecomment-1667453042
@tao12345666333 Sorry My bad it was a huge PEBCAK I did not see the #1777 😭
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] tao12345666333 commented on issue #1914: bug: Ingress controller crash loop when using ingress resources after restart
Posted by "tao12345666333 (via GitHub)" <gi...@apache.org>.
tao12345666333 commented on issue #1914:
URL: https://github.com/apache/apisix-ingress-controller/issues/1914#issuecomment-1664828397
This is a known issue and has been declared on the release page of v1.6.1.
https://github.com/apache/apisix-ingress-controller/issues/1777
> Known Issue
> A bug was discovered in v1.6.1, but it does not exist in version 1.6.0. We are actively working on fixing it and will address it in version 1.6.2. Please do not upgrade to 1.6.1 for now and continue using 1.6.0.
https://github.com/apache/apisix-ingress-controller/releases/tag/1.6.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] tao12345666333 commented on issue #1914: bug: Ingress controller crash loop when using ingress resources after restart
Posted by "tao12345666333 (via GitHub)" <gi...@apache.org>.
tao12345666333 commented on issue #1914:
URL: https://github.com/apache/apisix-ingress-controller/issues/1914#issuecomment-1667450541
https://github.com/apache/apisix-ingress-controller/issues/1914#issuecomment-1664828397
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org