You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2021/11/25 03:57:34 UTC
[GitHub] [apisix] jagerzhang opened a new issue #5604: bug: The latency of request is not stable, not as expected
jagerzhang opened a new issue #5604:
URL: https://github.com/apache/apisix/issues/5604
### Issue description
最近我们开始将生产环境的业务迁移到APISIX，迁移后有业务反馈延迟增大，实测了下，延迟确实存在抖动现象，在我们的生产环境中，接入APISIX的业务正常延时是25ms左右，但是抖动的时候延迟会超过200ms，最高到了800ms，对于延迟比较敏感的业务会受到影响。
不知道该如何定位。为了排除是我们生产环境网络或者APISIX配置的问题，我这边基于官方的example也测试了下，发现在本地网络环境下也存在抖动，正常应该是0ms，但是抖动的时候会达到10ms以上，不太符合APISIX 0.2ms延迟的预期。
具体测试用例和初步结果我已经上传到了github：https://github.com/jagerzhang/apisix-latency-test
辛苦APISIX研发专家帮忙看看这个是APISIX本身的问题还是其他什么因素影响的？ 如果这个测试用例10倍抖动不算问题的话，我们的生产环境抖动该如何定位？
### Environment
- apisix version (cmd: `apisix version`): centos-2.10.2
### Steps to reproduce
复现用例已整理上传，详见：https://github.com/jagerzhang/apisix-latency-test
### Actual result
```
cost: 13ms
cost: 11ms
cost: 12ms
cost: 15ms
cost: 11ms
cost: 11ms
cost: 11ms
cost: 11ms
cost: 13ms
cost: 14ms
cost: 14ms
cost: 11ms
cost: 13ms
cost: 13ms
cost: 12ms
Latency stats:
0ms: 9726
0~2ms: 191
2~5ms: 32
5~10ms: 28
>10ms: 22
```
### Error log
None
### Expected result
本地测试延迟应该在1ms左右比较正常，通过直接测试后端Nginx也可以发现延迟稳定很多。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] shuaijinchao commented on issue #5604: bug: The latency of request is not stable as expected
Posted by GitBox <gi...@apache.org>.
shuaijinchao commented on issue #5604:
URL: https://github.com/apache/apisix/issues/5604#issuecomment-978964939
From the data packet analysis, the time is indeed consumed on APISIX, we will try to reproduce it later. cc @tzssangglass
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] sandy420 edited a comment on issue #5604: bug: The latency of request is not stable as expected
Posted by GitBox <gi...@apache.org>.
sandy420 edited a comment on issue #5604:
URL: https://github.com/apache/apisix/issues/5604#issuecomment-979725978
> APISIX
@jagerzhang @tzssangglass me too,I mentioned issue before
[https://github.com/apache/apisix/issues/5500]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] jagerzhang commented on issue #5604: bug: The latency of request is not stable as expected
Posted by GitBox <gi...@apache.org>.
jagerzhang commented on issue #5604:
URL: https://github.com/apache/apisix/issues/5604#issuecomment-979756286
> > APISIX
>
> @jagerzhang @tzssangglass me too,I mentioned issue before #5500
Well, let's focus together
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] jagerzhang commented on issue #5604: bug: The latency of request is not stable as expected
Posted by GitBox <gi...@apache.org>.
jagerzhang commented on issue #5604:
URL: https://github.com/apache/apisix/issues/5604#issuecomment-978903474
> Have you tried to observe and analyze by capturing packets?
@shuaijinchao 刚抓包分析了下，找到其中耗时14ms的请求看了下详情，APISIX到Nginx之间耗时之后0.04ms，APISIX却耗时14ms：
完整抓包文件：
[start_test.cap.zip](https://github.com/apache/apisix/files/7600967/start_test.cap.zip)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] sandy420 edited a comment on issue #5604: bug: The latency of request is not stable as expected
Posted by GitBox <gi...@apache.org>.
sandy420 edited a comment on issue #5604:
URL: https://github.com/apache/apisix/issues/5604#issuecomment-979725978
> APISIX
@jagerzhang @tzssangglass me too,I mentioned issue before
https://github.com/apache/apisix/issues/5500
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] jagerzhang commented on issue #5604: bug: The latency of request is not stable as expected
Posted by GitBox <gi...@apache.org>.
jagerzhang commented on issue #5604:
URL: https://github.com/apache/apisix/issues/5604#issuecomment-986511335
it's Solved when the prometheus plugin was disabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] jagerzhang commented on issue #5604: bug: The latency of request is not stable as expected
Posted by GitBox <gi...@apache.org>.
jagerzhang commented on issue #5604:
URL: https://github.com/apache/apisix/issues/5604#issuecomment-1070366730
> @jagerzhang Have you solved this problem? I found that the official code has been merged, about the prometheus plugin deps
yes, solved, need to wait the next release：https://github.com/apache/apisix/pull/6315
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] jagerzhang edited a comment on issue #5604: bug: The latency of request is not stable as expected
Posted by GitBox <gi...@apache.org>.
jagerzhang edited a comment on issue #5604:
URL: https://github.com/apache/apisix/issues/5604#issuecomment-978903474
> Have you tried to observe and analyze by capturing packets?
@shuaijinchao 刚抓包分析了下，抓包命令：
`tcpdump -iany -nn port 9080 or port 8000 or port 8001 -vvv -w start_test.cap`
找到其中耗时14ms的请求看了下详情，APISIX到Nginx之间耗时仅0.04ms，APISIX 耗时有 14ms：
完整抓包文件：
[start_test.cap.zip](https://github.com/apache/apisix/files/7600967/start_test.cap.zip)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] sandy420 commented on issue #5604: bug: The latency of request is not stable as expected
Posted by GitBox <gi...@apache.org>.
sandy420 commented on issue #5604:
URL: https://github.com/apache/apisix/issues/5604#issuecomment-979725978
> APISIX
@jagerzhang @tzssangglass me too,I mentioned issue before
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] jagerzhang edited a comment on issue #5604: bug: The latency of request is not stable as expected
Posted by GitBox <gi...@apache.org>.
jagerzhang edited a comment on issue #5604:
URL: https://github.com/apache/apisix/issues/5604#issuecomment-978903474
> Have you tried to observe and analyze by capturing packets?
@shuaijinchao 刚抓包分析了下，抓包命令：
`tcpdump -iany -nn port 9080 or port 8000 or port 8001 -vvv -w start_test.cap`
找到其中耗时14ms的请求看了下详情，APISIX到Nginx之间耗时之后0.04ms，APISIX却耗时14ms：
完整抓包文件：
[start_test.cap.zip](https://github.com/apache/apisix/files/7600967/start_test.cap.zip)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] xianshun163 commented on issue #5604: bug: The latency of request is not stable as expected
Posted by GitBox <gi...@apache.org>.
xianshun163 commented on issue #5604:
URL: https://github.com/apache/apisix/issues/5604#issuecomment-1067669408
@jagerzhang Have you solved this problem? I found that the official code has been merged, about the prometheus plugin deps
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] shuaijinchao commented on issue #5604: bug: The latency of request is not stable as expected
Posted by GitBox <gi...@apache.org>.
shuaijinchao commented on issue #5604:
URL: https://github.com/apache/apisix/issues/5604#issuecomment-978882533
Have you tried to observe and analyze by capturing packets?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] sandy420 edited a comment on issue #5604: bug: The latency of request is not stable as expected
Posted by GitBox <gi...@apache.org>.
sandy420 edited a comment on issue #5604:
URL: https://github.com/apache/apisix/issues/5604#issuecomment-979725978
> APISIX
@jagerzhang @tzssangglass me too,I mentioned issue before
[https://github.com/apache/apisix/issues/5500](url)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org