You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2022/10/25 02:50:27 UTC
[GitHub] [apisix] jujiale opened a new issue, #8166: help request: proxy grpc sometimes occurs 502
jujiale opened a new issue, #8166:
URL: https://github.com/apache/apisix/issues/8166
### Description
hello:
we have two apisix env(such as A and B), when we use apisix to proxy grpc service, A always OK, but in B, when you first time send a request. It returns 502, but the following request all returns 200. and I also found that when we send a request in B, after a while，send request again, then the first request always return 502, the following return 200. I am confused of such thing. because A and B env apisix version and depends are same.
when 502 occurs. the following error occurs:
`
[error] 20620#20620: *327238 no connection data found for keepalive http2 connection while sending request to upstream
`
I want to get some idea from you. what causes it and how to solve it.
hope yours. thanks.
### Environment
- APISIX version (run `apisix version`): v2.12.0
- Operating system (run `uname -a`): the problem env is :Linux 3.10.0-1160.36.2.el7.x86_64. the good env is: Linux 3.10.0-957.21.3.el7.x86_64
- OpenResty / Nginx version (run `openresty -V` or `nginx -V`): nginx version: openresty/1.21.4.1
- etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`): 3.5.0
- APISIX Dashboard version, if relevant:
- Plugin runner version, for issues related to plugin runners:
- LuaRocks version, for installation issues (run `luarocks --version`):
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #8166: help request: proxy grpc sometimes occurs 502
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #8166:
URL: https://github.com/apache/apisix/issues/8166#issuecomment-1291598048
or disable `keepalive_pool` in upstream by set `idle_timeout`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] jujiale commented on issue #8166: help request: proxy grpc sometimes occurs 502
Posted by GitBox <gi...@apache.org>.
jujiale commented on issue #8166:
URL: https://github.com/apache/apisix/issues/8166#issuecomment-1292847346
> I thought this would be helpful to you: [argoproj/argo-cd#3896](https://github.com/argoproj/argo-cd/issues/3896) and [kubernetes/ingress-nginx#4095](https://github.com/kubernetes/ingress-nginx/issues/4095)
>
> Is your upstream service handling both grpc requests and http/1.1 requests on the same port?
Yes. our upstream service handling both grpc requests and http/1.1 requests on the same port.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #8166: help request: proxy grpc sometimes occurs 502
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #8166:
URL: https://github.com/apache/apisix/issues/8166#issuecomment-1291396472
1. what are routes and upstreams config info?
2. whether the route and the bound upstream which occurs 502 still proxy HTTP1.1/ requests？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] jujiale commented on issue #8166: help request: proxy grpc sometimes occurs 502
Posted by GitBox <gi...@apache.org>.
jujiale commented on issue #8166:
URL: https://github.com/apache/apisix/issues/8166#issuecomment-1291741996
> or disable `keepalive_pool` in upstream by set `idle_timeout` to 0.
I have tried this. the problem also exists.
I will continue to solve this problem. if I find the reason which caused the problem. I will comment it in here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] jujiale commented on issue #8166: help request: proxy grpc sometimes occurs 502
Posted by GitBox <gi...@apache.org>.
jujiale commented on issue #8166:
URL: https://github.com/apache/apisix/issues/8166#issuecomment-1291438818
> 1. what are routes and upstreams config info?
> 2. whether the route and the bound upstream which occurs 502 still proxy HTTP1.1/ requests？
@tzssangglass
1.
the following is our config info.
route config:
`
{
"uri": "/*",
"name": "xxx-xxx-xxx.grpc",
"priority": 100,
"methods": [
"GET",
"POST",
"PUT",
"DELETE",
"PATCH",
"HEAD",
"OPTIONS",
"CONNECT",
"TRACE"
],
"host": "xxx-xxx-xxx.grpc",
"upstream_id": "431317757666001848",
"status": 1
}
`
upstream config:
`
{
"nodes": [
{
"host": "172.xx.xxx.xxx",
"port": 20000,
"weight": 1
}
],
"timeout": {
"connect": 6,
"send": 6,
"read": 6
},
"type": "roundrobin",
"scheme": "grpc",
"pass_host": "pass",
"name": "xxx-xxx-xxx.grpc",
"keepalive_pool": {
"idle_timeout": 60,
"requests": 1000,
"size": 320
}
}
`
**and I think the config is not have problem. as I mentioned above, the same config in differenet env has different result. so I want to get some ideas from you. if you have some good suggestions. if some os config colud influence it**
2.
the route and the bound upstream not occurs 502 while proxy HTTP1.1/ requests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] jujiale commented on issue #8166: help request: proxy grpc sometimes occurs 502
Posted by GitBox <gi...@apache.org>.
jujiale commented on issue #8166:
URL: https://github.com/apache/apisix/issues/8166#issuecomment-1292894271
#https://github.com/kubernetes/ingress-nginx/issues/4836
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #8166: help request: proxy grpc sometimes occurs 502
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #8166:
URL: https://github.com/apache/apisix/issues/8166#issuecomment-1293469400
> Maybe APISIX needs to add the upstream scheme as a component of the keepalive pool name.
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #8166: help request: proxy grpc sometimes occurs 502
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #8166:
URL: https://github.com/apache/apisix/issues/8166#issuecomment-1291537247
> the route and the bound upstream not occurs 502 while proxy HTTP1.1/ requests
`"uri":"/*"` would result in a mix of grpc (HTTP/2) and HTTP/1.1 requests. I think you can create two routes, one of which hits only grpc requests and the other handles HTTP/1.1 requests.
Verify that it appears with this distinction 502.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] jujiale commented on issue #8166: help request: proxy grpc sometimes occurs 502
Posted by GitBox <gi...@apache.org>.
jujiale commented on issue #8166:
URL: https://github.com/apache/apisix/issues/8166#issuecomment-1291540713
> > the route and the bound upstream not occurs 502 while proxy HTTP1.1/ requests
>
> `"uri":"/*"` would result in a mix of grpc (HTTP/2) and HTTP/1.1 requests. I think you can create two routes, one of which hits only grpc requests and the other handles HTTP/1.1 requests.
>
> Verify that it appears with this distinction 502.
I do not think so. because we add host in mate rule. also apisix config.yaml open http2 port. we send grpc request with http2 port in apisix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander closed issue #8166: help request: proxy grpc sometimes occurs 502
Posted by GitBox <gi...@apache.org>.
spacewander closed issue #8166: help request: proxy grpc sometimes occurs 502
URL: https://github.com/apache/apisix/issues/8166
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #8166: help request: proxy grpc sometimes occurs 502
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #8166:
URL: https://github.com/apache/apisix/issues/8166#issuecomment-1291795131
I thought this would be helpful to you: https://github.com/argoproj/argo-cd/issues/3896 and https://github.com/kubernetes/ingress-nginx/issues/4095
Is your upstream service handling both grpc requests and http/1.1 requests on the same port?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] jujiale commented on issue #8166: help request: proxy grpc sometimes occurs 502
Posted by GitBox <gi...@apache.org>.
jujiale commented on issue #8166:
URL: https://github.com/apache/apisix/issues/8166#issuecomment-1292886601
I find as in this issue said. keepalive might be the reason which caused the problem.
[https://github.com/kubernetes/ingress-nginx/issues/4836#issuecomment-871342543](url)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] jacobslei commented on issue #8166: help request: proxy grpc sometimes occurs 502
Posted by GitBox <gi...@apache.org>.
jacobslei commented on issue #8166:
URL: https://github.com/apache/apisix/issues/8166#issuecomment-1311226498
> > Maybe APISIX needs to add the upstream scheme as a component of the keepalive pool name.
>
> +1
+1, full of expectation. So what's the plannning that APISIX will support this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #8166: help request: proxy grpc sometimes occurs 502
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #8166:
URL: https://github.com/apache/apisix/issues/8166#issuecomment-1291601863
> I do not think so. because we add host in mate rule. also apisix config.yaml open http2 port. we send grpc request with http2 port in apisix.
You can try sending an HTTP/1.1 request to the http2 port, as long as the route and upstream are configured, it should be able to be forwarded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] jujiale commented on issue #8166: help request: proxy grpc sometimes occurs 502
Posted by GitBox <gi...@apache.org>.
jujiale commented on issue #8166:
URL: https://github.com/apache/apisix/issues/8166#issuecomment-1291798987
> I thought this would be helpful to you: [argoproj/argo-cd#3896](https://github.com/argoproj/argo-cd/issues/3896) and [kubernetes/ingress-nginx#4095](https://github.com/kubernetes/ingress-nginx/issues/4095)
>
> Is your upstream service handling both grpc requests and http/1.1 requests on the same port?
I will have a look. thank you very much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on issue #8166: help request: proxy grpc sometimes occurs 502
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #8166:
URL: https://github.com/apache/apisix/issues/8166#issuecomment-1293276941
Maybe APISIX needs to add the upstream scheme as a component of the keepalive pool name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] soulbird commented on issue #8166: help request: proxy grpc sometimes occurs 502
Posted by GitBox <gi...@apache.org>.
soulbird commented on issue #8166:
URL: https://github.com/apache/apisix/issues/8166#issuecomment-1313050679
> > > Maybe APISIX needs to add the upstream scheme as a component of the keepalive pool name.
> >
> >
> > +1
>
> +1, full of expectation. So what's the plannning that APISIX will support this issue?
I'll fix it soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org