You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2021/09/07 03:28:48 UTC
[GitHub] [apisix-ingress-controller] kika-jiaxudong opened a new issue #668: bug: 2 apisix-ingress-controller,one is normal and another always is starting
kika-jiaxudong opened a new issue #668:
URL: https://github.com/apache/apisix-ingress-controller/issues/668
### Issue description
2 apisix-ingress-controller replicas,one is norma and another always is starting
### Environment
helm installation
Chart Version:
apisix-0.3.6
Apisix Version:
2.7.0
### Steps to reproduce
installation by helm
set the replicas of apisix-ingress-controller to 2
### Actual result
found one apisix-ingress-controller replica is normal and another is been rebooting.
describe pod info:
" State: Running
Started: Mon, 06 Sep 2021 22:36:55 +0000
Last State: Terminated
Reason: Error
Exit Code: 2
Started: Mon, 06 Sep 2021 16:36:54 +0000
Finished: Mon, 06 Sep 2021 22:36:54 +0000
Ready: True
Restart Count: 16
"
### Error log
[GIN] 2021/09/07 - 10:33:12 | 200 | 37.262µs | 192.168.94.37 | GET "/healthz"
2021-09-07T10:33:13+08:00 info ingress/controller.go:530 service kube-system/kube-controller-manager not found
2021-09-07T10:33:13+08:00 info ingress/controller.go:530 service kube-system/kube-scheduler not found
[GIN] 2021/09/07 - 10:33:14 | 200 | 40.64µs | 192.168.94.37 | GET "/healthz"
2021-09-07T10:33:15+08:00 info ingress/controller.go:530 service kube-system/kube-controller-manager not found
2021-09-07T10:33:15+08:00 info ingress/controller.go:530 service kube-system/kube-scheduler not found
2021-09-07T10:33:17+08:00 info ingress/controller.go:530 service kube-system/kube-controller-manager not found
2021-09-07T10:33:17+08:00 info ingress/controller.go:530 service kube-system/kube-scheduler not found
2021-09-07T10:33:19+08:00 info ingress/controller.go:530 service kube-system/kube-controller-manager not found
2021-09-07T10:33:19+08:00 info ingress/controller.go:530 service kube-system/kube-scheduler not found
2021-09-07T10:33:21+08:00 info ingress/controller.go:530 service kube-system/kube-controller-manager not found
2021-09-07T10:33:21+08:00 info ingress/controller.go:530 service kube-system/kube-scheduler not found
[GIN] 2021/09/07 - 10:33:22 | 200 | 44.393µs | 192.168.94.37 | GET "/healthz"
2021-09-07T10:33:23+08:00 info ingress/controller.go:530 service kube-system/kube-controller-manager not found
2021-09-07T10:33:23+08:00 info ingress/controller.go:530 service kube-system/kube-scheduler not found
[GIN] 2021/09/07 - 10:33:24 | 200 | 42.71µs | 192.168.94.37 | GET "/healthz"
2021-09-07T10:33:25+08:00 info ingress/controller.go:530 service kube-system/kube-controller-manager not found
2021-09-07T10:33:25+08:00 info ingress/controller.go:530 service kube-system/kube-scheduler not found
2021-09-07T10:33:27+08:00 info ingress/controller.go:530 service kube-system/kube-controller-manager not found
2021-09-07T10:33:27+08:00 info ingress/controller.go:530 service kube-system/kube-scheduler not found
2021-09-07T10:33:29+08:00 info ingress/controller.go:530 service kube-system/kube-controller-manager not found
2021-09-07T10:33:29+08:00 info ingress/controller.go:530 service kube-system/kube-scheduler not found
E0907 10:33:31.221087 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.21.1/tools/cache/reflector.go:167: Failed to watch *v1.Ingress: failed to list *v1.Ingress: the server could not find the requested resource
2021-09-07T10:33:31+08:00 info ingress/controller.go:530 service kube-system/kube-controller-manager not found
2021-09-07T10:33:31+08:00 info ingress/controller.go:530 service kube-system/kube-scheduler not found
[GIN] 2021/09/07 - 10:33:32 | 200 | 38.322µs | 192.168.94.37 | GET "/healthz"
### Expected result
2 replicas normal
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] kika-jiaxudong commented on issue #668: bug: 2 apisix-ingress-controller,one is normal and another always is starting
Posted by GitBox <gi...@apache.org>.
kika-jiaxudong commented on issue #668:
URL: https://github.com/apache/apisix-ingress-controller/issues/668#issuecomment-920673338
> logs like "service xxx not found" are harmless, they were logged when endpoints with the same name were changed but the service was not found.
>
> ```
> E0915 20:10:27.750659 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.21.1/tools/cache/reflector.go:167: Failed to watch *v1.Ingress: failed to list *v1.Ingress: the server could not find the requested resource
> ```
>
> is also not a fatal one. Have you ever investigate the pod status is `Error` or `CrashLoopBackoff`?
wow,I find the real error like that:
<img width="1436" alt="image" src="https://user-images.githubusercontent.com/83063375/133571531-3a05c5f2-b00c-47e3-b99b-7e109cd8666c.png">
是它导致了apisix-ingress-controller主节点一直重启，我的apisixroute的yaml文件是这样的，请帮我检查下有什么问题，很奇怪，为什么通过apisix的流量好像是没有问题的，但是在apisix-ingress-controller里日志尽然显示不支持这样的资源？
<img width="438" alt="image" src="https://user-images.githubusercontent.com/83063375/133572694-2cc46c29-8b4f-47b9-8200-5c215c904c3f.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] gxthrj commented on issue #668: bug: 2 apisix-ingress-controller,one is normal and another always is starting
Posted by GitBox <gi...@apache.org>.
gxthrj commented on issue #668:
URL: https://github.com/apache/apisix-ingress-controller/issues/668#issuecomment-917618973
I had deploy ingress with 2 instances using the same configurations with you, now 2 days later, there is no restart.
Please give more information about your environment. For example, your route configuration with ApisixRoute resource.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] kika-jiaxudong commented on issue #668: bug: 2 apisix-ingress-controller,one is normal and another always is starting
Posted by GitBox <gi...@apache.org>.
kika-jiaxudong commented on issue #668:
URL: https://github.com/apache/apisix-ingress-controller/issues/668#issuecomment-920529369
<img width="938" alt="image" src="https://user-images.githubusercontent.com/83063375/133540118-2d31539c-c599-4ec4-a41f-c6297b439def.png">
<img width="1439" alt="image" src="https://user-images.githubusercontent.com/83063375/133540305-094295b5-360f-44bf-b6ee-90778ddc6080.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] tokers commented on issue #668: bug: 2 apisix-ingress-controller,one is normal and another always is starting
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #668:
URL: https://github.com/apache/apisix-ingress-controller/issues/668#issuecomment-920492774
logs like "service xxx not found" are harmless, they were logged when endpoints with the same name were changed but the service was not found.
```
E0915 20:10:27.750659 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.21.1/tools/cache/reflector.go:167: Failed to watch *v1.Ingress: failed to list *v1.Ingress: the server could not find the requested resource
```
is also not a fatal one. Have you ever investigate the pod status is `Error` or `CrashLoopBackoff`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] nic-chen commented on issue #668: bug: 2 apisix-ingress-controller,one is normal and another always is starting
Posted by GitBox <gi...@apache.org>.
nic-chen commented on issue #668:
URL: https://github.com/apache/apisix-ingress-controller/issues/668#issuecomment-943185447
ping @gxthrj @tao12345666333
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] kika-jiaxudong commented on issue #668: bug: 2 apisix-ingress-controller,one is normal and another always is starting
Posted by GitBox <gi...@apache.org>.
kika-jiaxudong commented on issue #668:
URL: https://github.com/apache/apisix-ingress-controller/issues/668#issuecomment-919956962
> Please provide your deployment environment information (k8s version etc.)
k8s version: v1.18.20 on eks(aws eks)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] tokers commented on issue #668: bug: 2 apisix-ingress-controller,one is normal and another always is starting
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #668:
URL: https://github.com/apache/apisix-ingress-controller/issues/668#issuecomment-920900244
It seems that the v2beta1 versioned ApisixRoute are not installed in your version, would you like to upgrade your Apache APISIX Ingress Controller version and see what will happen?
But another thing that I concern is that these logs won't result in the restart of the program, the error is not propagated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] kika-jiaxudong commented on issue #668: bug: 2 apisix-ingress-controller,one is normal and another always is starting
Posted by GitBox <gi...@apache.org>.
kika-jiaxudong commented on issue #668:
URL: https://github.com/apache/apisix-ingress-controller/issues/668#issuecomment-925658211
> > > It seems that the v2beta1 versioned ApisixRoute are not installed in your version, would you like to upgrade your Apache APISIX Ingress Controller version and see what will happen?
> > > But another thing that I concern is that these logs won't result in the restart of the program, the error is not propagated.
> >
> >
> > Actural, I had installed apisix apiversion v2beta1,like this:
> > <img alt="image" width="562" src="https://user-images.githubusercontent.com/83063375/133734524-ece4e8c2-67a1-4f49-a566-29b2a7f018de.png">
> > 我能确定每次当我修改apisixroute的yaml文件，然后再kubectl apply -f执行它的时候，apisix-ingress-controller主节点会变为Error的状态，然后过一会会重启，再次变为running的状态
> > 我想请您帮我确认一下，在我的apisixroute的yaml中，最开始一行是apiVersion: apisix.apache.org/v1
> > 但是在metadata.manageddFields.apiVersion是
> > apisix.apache.org/v2beta1
> > 请问哪一个是生效的，
> > 其实在我的yaml文件中只是设置了apisix.apache.org/v2beta1，但是kubectl apply -f之后变为了这样，如下：
> > 
>
> OK, let me reproduce it.
hello,have you reproduce it?If the same error occurred？
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] tokers commented on issue #668: bug: 2 apisix-ingress-controller,one is normal and another always is starting
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #668:
URL: https://github.com/apache/apisix-ingress-controller/issues/668#issuecomment-914337494
The description is a bit confusing, where is the description of the starting pod?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] kika-jiaxudong commented on issue #668: bug: 2 apisix-ingress-controller,one is normal and another always is starting
Posted by GitBox <gi...@apache.org>.
kika-jiaxudong commented on issue #668:
URL: https://github.com/apache/apisix-ingress-controller/issues/668#issuecomment-920527606
> logs like "service xxx not found" are harmless, they were logged when endpoints with the same name were changed but the service was not found.
>
> ```
> E0915 20:10:27.750659 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.21.1/tools/cache/reflector.go:167: Failed to watch *v1.Ingress: failed to list *v1.Ingress: the server could not find the requested resource
> ```
>
> is also not a fatal one. Have you ever investigate the pod status is `Error` or `CrashLoopBackoff`?
The pod status is always running, but can get the pod info via "kubectl describe pod",I get that like this:
<img width="1096" alt="image" src="https://user-images.githubusercontent.com/83063375/133539804-90fbf831-40e8-48d8-8f2c-05cdb61b08ab.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] kika-jiaxudong commented on issue #668: bug: 2 apisix-ingress-controller,one is normal and another always is starting
Posted by GitBox <gi...@apache.org>.
kika-jiaxudong commented on issue #668:
URL: https://github.com/apache/apisix-ingress-controller/issues/668#issuecomment-921539549
> It seems that the v2beta1 versioned ApisixRoute are not installed in your version, would you like to upgrade your Apache APISIX Ingress Controller version and see what will happen?
>
> But another thing that I concern is that these logs won't result in the restart of the program, the error is not propagated.
Actural, I had installed apisix apiversion v2beta1,like this:
<img width="562" alt="image" src="https://user-images.githubusercontent.com/83063375/133734524-ece4e8c2-67a1-4f49-a566-29b2a7f018de.png">
我能确定每次当我修改apisixroute的yaml文件，然后再kubectl apply -f执行它的时候，apisix-ingress-controller主节点会变为Error的状态，然后过一会会重启，再次变为running的状态
我想请您帮我确认一下，在我的apisixroute的yaml中，最开始一行是apiVersion: apisix.apache.org/v1
但是在metadata.manageddFields.apiVersion是
apisix.apache.org/v2beta1
请问哪一个是生效的，
其实在我的yaml文件中只是设置了apisix.apache.org/v2beta1，但是kubectl apply -f之后变为了这样，如下：
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] kika-jiaxudong commented on issue #668: bug: 2 apisix-ingress-controller,one is normal and another always is starting
Posted by GitBox <gi...@apache.org>.
kika-jiaxudong commented on issue #668:
URL: https://github.com/apache/apisix-ingress-controller/issues/668#issuecomment-919960854
> I had deploy ingress with 2 instances using the same configurations with you, now 2 days later, there is no restart.
> 
>
> Please give more information about your environment. For example, your route configuration with ApisixRoute resource.
yes,when i just install apisix ingress controller with 2 replicas,it don't always reboot,
but when i create apisixroute via kubectl,it begin to always rebooting,
apisixroute like this:
"
apiVersion: apisix.apache.org/v2beta1
kind: ApisixRoute
metadata:
name: backend-operation-platform
namespace: backend-operation-platform-test
spec:
http:
- backend:
serviceName: backend-operation-platform
servicePort: 8082
match:
hosts:
- operation-dev.kele.com
paths:
- /v1*
- /global*
name: backend-operation-platform
plugins:
- config:
whitelist:
- 192.168.0.0/16
enable: true
name: ip-restriction
"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] kika-jiaxudong commented on issue #668: bug: 2 apisix-ingress-controller,one is normal and another always is starting
Posted by GitBox <gi...@apache.org>.
kika-jiaxudong commented on issue #668:
URL: https://github.com/apache/apisix-ingress-controller/issues/668#issuecomment-919967904
> The description is a bit confusing, where is the description of the starting pod?
主要是这些错误日志，在一直重启的ingress-controller的一个pod上，apisix-ingress-controller有2个副本，另一个正常，一直没有重启过，但是还有一个pod一直在重启，错误日志如下：
2021-09-15T20:10:26+08:00 info ingress/controller.go:530 service kube-system/kube-scheduler not found
2021-09-15T20:10:27+08:00 info ingress/controller.go:530 service kube-system/kube-controller-manager not found
E0915 20:10:27.750659 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.21.1/tools/cache/reflector.go:167: Failed to watch *v1.Ingress: failed to list *v1.Ingress: the server could not find the requested resource
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] tao12345666333 commented on issue #668: bug: 2 apisix-ingress-controller,one is normal and another always is starting
Posted by GitBox <gi...@apache.org>.
tao12345666333 commented on issue #668:
URL: https://github.com/apache/apisix-ingress-controller/issues/668#issuecomment-919586471
Please provide your deployment environment information (k8s version etc.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] tokers commented on issue #668: bug: 2 apisix-ingress-controller,one is normal and another always is starting
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #668:
URL: https://github.com/apache/apisix-ingress-controller/issues/668#issuecomment-921628208
> > It seems that the v2beta1 versioned ApisixRoute are not installed in your version, would you like to upgrade your Apache APISIX Ingress Controller version and see what will happen?
> > But another thing that I concern is that these logs won't result in the restart of the program, the error is not propagated.
>
> Actural, I had installed apisix apiversion v2beta1,like this:
> <img alt="image" width="562" src="https://user-images.githubusercontent.com/83063375/133734524-ece4e8c2-67a1-4f49-a566-29b2a7f018de.png">
>
> 我能确定每次当我修改apisixroute的yaml文件，然后再kubectl apply -f执行它的时候，apisix-ingress-controller主节点会变为Error的状态，然后过一会会重启，再次变为running的状态
>
> 我想请您帮我确认一下，在我的apisixroute的yaml中，最开始一行是apiVersion: apisix.apache.org/v1
>
> 但是在metadata.manageddFields.apiVersion是
> apisix.apache.org/v2beta1
>
> 请问哪一个是生效的，
>
> 其实在我的yaml文件中只是设置了apisix.apache.org/v2beta1，但是kubectl apply -f之后变为了这样，如下：
>
> 
OK, let me reproduce it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix-ingress-controller] kika-jiaxudong commented on issue #668: bug: 2 apisix-ingress-controller,one is normal and another always is starting
Posted by GitBox <gi...@apache.org>.
kika-jiaxudong commented on issue #668:
URL: https://github.com/apache/apisix-ingress-controller/issues/668#issuecomment-920528083
<img width="677" alt="image" src="https://user-images.githubusercontent.com/83063375/133539966-0171009f-3d9a-4a1c-b209-2671c54b05b4.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org