You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2021/02/02 13:38:51 UTC
[GitHub] [apisix] xyz101121 opened a new issue #3498: apisix access have 500 status code
xyz101121 opened a new issue #3498:
URL: https://github.com/apache/apisix/issues/3498
when i turn on healthcheck, they are too many 500 ,error log:
when i turn off healthcheck ,500 status is Very few,error log：
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] panhow commented on issue #3498: apisix access have 500 status code
Posted by GitBox <gi...@apache.org>.
panhow commented on issue #3498:
URL: https://github.com/apache/apisix/issues/3498#issuecomment-994872391
i got the same problem.
reproduce steps:
1. configure a upstream with healthcheck
2. use `wrk` on 3 nodes, 100 concurrency connections to reqeust the same route
guess this promblem occurs cuz active healthcheck will active after first request to upstream, before heallthcheck active, get the initialized healthy nodes list, checker return `nil` to balancer_by_lua
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] xyz101121 commented on issue #3498: apisix access have 500 status code
Posted by GitBox <gi...@apache.org>.
xyz101121 commented on issue #3498:
URL: https://github.com/apache/apisix/issues/3498#issuecomment-772302556
This problem is not obligatory, it only appears occasionally when there are a large number of requests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] Firstsawyou commented on issue #3498: apisix access have 500 status code
Posted by GitBox <gi...@apache.org>.
Firstsawyou commented on issue #3498:
URL: https://github.com/apache/apisix/issues/3498#issuecomment-771655724
Okay, let me check.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] Firstsawyou commented on issue #3498: apisix access have 500 status code
Posted by GitBox <gi...@apache.org>.
Firstsawyou commented on issue #3498:
URL: https://github.com/apache/apisix/issues/3498#issuecomment-772360807
> But only a large number of requests have this problem, and I cannot reproduce it through the curl command
The `curl` command is not necessary, it only needs to be able to see the response header of the requested response data.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] Firstsawyou commented on issue #3498: apisix access have 500 status code
Posted by GitBox <gi...@apache.org>.
Firstsawyou commented on issue #3498:
URL: https://github.com/apache/apisix/issues/3498#issuecomment-771655724
Okay, let me check.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] xyz101121 commented on issue #3498: apisix access have 500 status code
Posted by GitBox <gi...@apache.org>.
xyz101121 commented on issue #3498:
URL: https://github.com/apache/apisix/issues/3498#issuecomment-771641244
apisix is docker , image is apache/apisix:2.2-alpine
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] xyz101121 commented on issue #3498: apisix access have 500 status code
Posted by GitBox <gi...@apache.org>.
xyz101121 commented on issue #3498:
URL: https://github.com/apache/apisix/issues/3498#issuecomment-772351235
But only a large number of requests have this problem, and I cannot reproduce it through the curl command
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] xyz101121 commented on issue #3498: apisix access have 500 status code
Posted by GitBox <gi...@apache.org>.
xyz101121 commented on issue #3498:
URL: https://github.com/apache/apisix/issues/3498#issuecomment-772392896
OK，I can only print in the log
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] xyz101121 commented on issue #3498: apisix access have 500 status code
Posted by GitBox <gi...@apache.org>.
xyz101121 commented on issue #3498:
URL: https://github.com/apache/apisix/issues/3498#issuecomment-772269570
this is my apisix route config:
```
{
"modifiedIndex": 16272,
"createdIndex": 3099,
"key": "/apisix/routes/338984743892681423",
"value": {
"labels": {
"domain": "xxxxx.cn",
"API_VERSION": "1"
},
"uris": [
"/*"
],
"methods": [
"GET",
"HEAD",
"POST"
],
"hosts": [
"www.xxxxx.com",
"www.xxxxx.cn",
"xxxxx.com",
"xxxxxx.cn"
],
"create_time": 1611580252,
"id": "338984743892681423",
"service_id": "338981951442518735",
"upstream_id": "338952520397226703",
"status": 1,
"desc": "/",
"vars": [
[
"scheme",
"==",
"http"
]
],
"update_time": 1612264551,
"name": "www_xxxxxxx_cn_http"
}
}
```
this is my apisix service config:
```
{
"count": "1",
"action": "get",
"node": {
"key": "/apisix/services/338981951442518735",
"value": {
"update_time": 1612264423,
"name": "server1_server2_server3",
"plugins": {
"echo": {
"disable": true,
"before_body": "before the body modification "
}
},
"upstream_id": "338952520397226703",
"create_time": 1611578588,
"id": "338981951442518735"
}
}
}
```
this is my apisix upstream config:
```
{
"count": "1",
"action": "get",
"node": {
"key": "/apisix/upstreams/338952520397226703",
"value": {
"update_time": 1612272346,
"name": "web1_web2_web3_http",
"pass_host": "pass",
"create_time": 1611561045,
"timeout": {
"send": 6000,
"read": 6000,
"connect": 6000
},
"nodes": [
{
"port": 80,
"weight": 1,
"host": "172.16.0.237"
},
{
"port": 80,
"weight": 1,
"host": "172.16.0.185"
},
{
"port": 80,
"weight": 1,
"host": "172.16.0.172"
}
],
"desc": "关闭健康检测",
"type": "roundrobin",
"id": "338952520397226703"
}
}
}
```
Thanks your help!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] membphis commented on issue #3498: apisix access have 500 status code
Posted by GitBox <gi...@apache.org>.
membphis commented on issue #3498:
URL: https://github.com/apache/apisix/issues/3498#issuecomment-772426689
@xyz101121 if possible, please make a try with the master branch or 2.3 branch. I can not sure if this bug is fixed already.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] membphis commented on issue #3498: apisix access have 500 status code
Posted by GitBox <gi...@apache.org>.
membphis commented on issue #3498:
URL: https://github.com/apache/apisix/issues/3498#issuecomment-995479146
@panhow this issue is old.
you can create a new issue with your reproduce steps(apisix version, os version, etc.)
https://github.com/apache/apisix/issues/new?assignees=&labels=&template=bug_report.yml&title=bug%3A+
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on issue #3498: apisix access have 500 status code
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #3498:
URL: https://github.com/apache/apisix/issues/3498#issuecomment-772468051
> But only a large number of requests have this problem, and I cannot reproduce it through the curl command
Do you have the tcpdump utility in the environment? If any, try to capture some packets.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] xyz101121 commented on issue #3498: apisix access have 500 status code
Posted by GitBox <gi...@apache.org>.
xyz101121 commented on issue #3498:
URL: https://github.com/apache/apisix/issues/3498#issuecomment-772254437
now I turn off health check ,But 500 status code still exists，
this is my apisix route config:
this is my apisix service config:
this is my apisix upstream config:
this is apisix request access. log for 500 status code
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] Firstsawyou commented on issue #3498: apisix access have 500 status code
Posted by GitBox <gi...@apache.org>.
Firstsawyou commented on issue #3498:
URL: https://github.com/apache/apisix/issues/3498#issuecomment-771821333
Can't reproduce, maybe you need to provide your minimum steps to reproduce.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] Firstsawyou commented on issue #3498: apisix access have 500 status code
Posted by GitBox <gi...@apache.org>.
Firstsawyou commented on issue #3498:
URL: https://github.com/apache/apisix/issues/3498#issuecomment-772299489
Based on the information you provided, your problem cannot be reproduced. And there is no health check in your configuration. Here is my test:
environment： docker , image is apache/apisix:2.2-alpine
1、add route:
```
curl http://127.0.0.1:9080/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
"labels": {
"domain": "test.cn",
"API_VERSION": "1"
},
"uris": [
"/*"
],
"methods": [
"GET",
"HEAD",
"POST"
],
"service_id": "1",
"upstream_id": "1",
"status": 1,
"desc": "/",
"vars": [
[
"scheme",
"==",
"http"
]
],
"name": "www_xxxxxxx_cn_http"
}'
```
2、add service:
```
curl http://127.0.0.1:9080/apisix/admin/services/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
"name": "server1_server2_server3",
"plugins": {
"echo": {
"disable": true,
"before_body": "before the body modification "
}
},
"upstream_id": "1"
}'
```
3、add upstream
```
curl http://127.0.0.1:9080/apisix/admin/upstreams/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -i -X PUT -d '
{
"name": "web1_web2_web3_http",
"pass_host": "pass",
"timeout": {
"send": 6000,
"read": 6000,
"connect": 6000
},
"nodes": [
{
"port": 80,
"weight": 1,
"host": "httpbin.org"
},
{
"port": 80,
"weight": 1,
"host": "172.16.0.172"
}
],
"desc": "关闭健康检测",
"type": "roundrobin"
}'
```
4、test request:
```
$ curl http://127.0.0.1:9080/get -i
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 299
Connection: keep-alive
Date: Wed, 03 Feb 2021 07:26:10 GMT
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Server: APISIX/2.2
{
"args": {},
"headers": {
"Accept": "*/*",
"Host": "127.0.0.1",
"User-Agent": "curl/7.67.0",
"X-Amzn-Trace-Id": "Root=1-601a5012-7a9877a178fe768b63206032",
"X-Forwarded-Host": "127.0.0.1"
},
"origin": "127.0.0.1, 121.41.116.83",
"url": "http://127.0.0.1/get"
}
```
```
$ curl http://127.0.0.1:9080/index.hmtl -i
HTTP/1.1 404 NOT FOUND
Content-Type: text/html; charset=utf-8
Content-Length: 233
Connection: keep-alive
Date: Wed, 03 Feb 2021 07:30:20 GMT
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Server: APISIX/2.2
........
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] Firstsawyou commented on issue #3498: apisix access have 500 status code
Posted by GitBox <gi...@apache.org>.
Firstsawyou commented on issue #3498:
URL: https://github.com/apache/apisix/issues/3498#issuecomment-772332757
> they are status code from my apisix log
> [root@ecs-551a apisix_log]# grep 'GET|POST' access.log |awk '{print $10}'|sort -rn|uniq -c
>
> ```
> 37 502
> 1744 500
> 314 499
> 2 408
> 17238 404
> 166 403
> 26 401
> 101 400
> 1003 304
> 18417 302
> 30 301
> 75 206
> 7 201
> 611597 200
> ```
You should check whether the response data contains the header information of `X-APISIX-Upstream-Status` in the 500 request. This header information can help us troubleshoot the problem.
For example:
```
$ curl http://127.0.0.1:9080/hello -i
HTTP/1.1 500 Internal Server Error
Date: Wed, 03 Feb 2021 01:35:00 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 154
Connection: keep-alive
Server: APISIX/2.2
X-APISIX-Upstream-Status: 500, 500
......
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] xyz101121 commented on issue #3498: apisix access have 500 status code
Posted by GitBox <gi...@apache.org>.
xyz101121 commented on issue #3498:
URL: https://github.com/apache/apisix/issues/3498#issuecomment-771641244
apisix is docker , image is apache/apisix:2.2-alpine
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] xyz101121 commented on issue #3498: apisix access have 500 status code
Posted by GitBox <gi...@apache.org>.
xyz101121 commented on issue #3498:
URL: https://github.com/apache/apisix/issues/3498#issuecomment-772323349
they are status code from my apisix log
[root@ecs-551a apisix_log]# grep 'GET\|POST' access.log |awk '{print $10}'|sort -rn|uniq -c
```
37 502
1744 500
314 499
2 408
17238 404
166 403
26 401
101 400
1003 304
18417 302
30 301
75 206
7 201
611597 200
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #3498: apisix access have 500 status code
Posted by GitBox <gi...@apache.org>.
spacewander commented on issue #3498:
URL: https://github.com/apache/apisix/issues/3498#issuecomment-772132959
How you configure the health check?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] membphis edited a comment on issue #3498: apisix access have 500 status code
Posted by GitBox <gi...@apache.org>.
membphis edited a comment on issue #3498:
URL: https://github.com/apache/apisix/issues/3498#issuecomment-995479146
@panhow this issue is old.
you can create a new issue with your reproduce steps(apisix version, os version, etc.)
https://github.com/apache/apisix/issues/new?assignees=&labels=&template=bug_report.yml&title=bug%3A+
many thx.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] membphis commented on issue #3498: apisix access have 500 status code
Posted by GitBox <gi...@apache.org>.
membphis commented on issue #3498:
URL: https://github.com/apache/apisix/issues/3498#issuecomment-996392247
ok, got it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] membphis commented on issue #3498: apisix access have 500 status code
Posted by GitBox <gi...@apache.org>.
membphis commented on issue #3498:
URL: https://github.com/apache/apisix/issues/3498#issuecomment-772422999
> This problem is not obligatory, it only appears occasionally when there are a large number of requests
Can you provide more information about how to reproduce this issue? @xyz101121
The only configuration is not enough.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] panhow commented on issue #3498: apisix access have 500 status code
Posted by GitBox <gi...@apache.org>.
panhow commented on issue #3498:
URL: https://github.com/apache/apisix/issues/3498#issuecomment-995632845
> @panhow this issue is old.
>
> you can create a new issue with your reproduce steps(apisix version, os version, etc.)
>
> https://github.com/apache/apisix/issues/new?assignees=&labels=&template=bug_report.yml&title=bug%3A+
>
> many thx.
thx for replied, after i reproduce on latest apisix, i will create a new issue with reproduce steps.
i tried on ``2.10`` this morning, but it didn't reproduce again, i dont known it has been fixed or not enough pressure for the server.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] membphis closed issue #3498: apisix access have 500 status code
Posted by GitBox <gi...@apache.org>.
membphis closed issue #3498:
URL: https://github.com/apache/apisix/issues/3498
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #3498: apisix access have 500 status code
Posted by GitBox <gi...@apache.org>.
spacewander commented on issue #3498:
URL: https://github.com/apache/apisix/issues/3498#issuecomment-772291897
> How you configure the health check?
What is your original health check configuration?
Your provided data only contains the upstream without the checks field.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org