You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2022/02/14 06:57:56 UTC
[GitHub] [apisix] tzssangglass opened a new pull request #6315: chore: update ngxin-lua-prometheus verison
tzssangglass opened a new pull request #6315:
URL: https://github.com/apache/apisix/pull/6315
Closes #5755
### What this PR does / why we need it:
<!--- Why is this change required? What problem does it solve? -->
<!--- If it fixes an open issue, please link to the issue here. -->
### Pre-submission checklist:
<!--
Please follow the PR manners:
1. Use Draft if the PR is not ready to be reviewed
2. Test is required for the feat/fix PR, unless you have a good reason
3. Doc is required for the feat PR
4. Use a new commit to resolve review instead of `push -f`
5. If you need to resolve merge conflicts after the PR is reviewed, please merge master but do not rebase
6. Use "request review" to notify the reviewer once you have resolved the review
7. Only reviewer can click "Resolve conversation" to mark the reviewer's review resolved
-->
* [x] Did you explain what problem does this PR solve? Or what new features have been added?
* [ ] Have you added corresponding test cases?
* [ ] Have you modified the corresponding document?
* [x] Is this PR backward compatible? **If it is not backward compatible, please discuss on the [mailing list](https://github.com/apache/apisix/tree/master#community) first**
The following is a simple test
prepare
1. add prometheus plugin
```shell
curl -i http://127.0.0.1:9080/apisix/adminglobal_rules/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
"plugins": {
"prometheus": {}
}
}'
```
2. add rouet
```shell
curl -i http://127.0.0.1:9080/apisix/adminroutes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
"upstream": {
"nodes": {
"127.0.0.1:1980": 1
},
"type": "roundrobin"
},
"uri": "/hello"
}'
```
3. wrk2
```shell
# mock client request
wrk -t4 -c100 -d90s -R5000 --u_latency http://127.0.0.1:9080/hello
# mock prometheus collection
wrk -t4 -c100 -d100s -R10 --u_latency http://127.0.0.1:9091/apisix/prometheus/metrics
```
#### before update prometheus version:
*request latency*
```
Latency Distribution (HdrHistogram - Uncorrected Latency (measured without taking delayed starts into account))
50.000% 415.00us
75.000% 594.00us
90.000% 0.87ms
99.000% 4.95ms
99.900% 17.26ms
99.990% 33.53ms
99.999% 40.96ms
100.000% 46.69ms
```
*prometheus latency*
```
Latency Distribution (HdrHistogram - Uncorrected Latency (measured without taking delayed starts into account))
50.000% 9.73ms
75.000% 17.36ms
90.000% 27.34ms
99.000% 38.85ms
99.900% 41.50ms
99.990% 43.49ms
99.999% 43.49ms
100.000% 43.49ms
```
flamegraph
#### after update prometheus version:
*request latency*
```
Latency Distribution (HdrHistogram - Uncorrected Latency (measured without taking delayed starts into account))
50.000% 605.00us
75.000% 0.89ms
90.000% 1.26ms
99.000% 2.58ms
99.900% 9.80ms
99.990% 14.20ms
99.999% 18.74ms
100.000% 19.65ms
```
*prometheus latency*
```
Latency Distribution (HdrHistogram - Uncorrected Latency (measured without taking delayed starts into account))
50.000% 1.67ms
75.000% 2.10ms
90.000% 2.86ms
99.000% 6.89ms
99.900% 10.61ms
99.990% 10.67ms
99.999% 10.67ms
100.000% 10.67ms
```
flamegraph
We can assume that the impact on request latency when prometheus collects information is significantly reduced after the update.
#### why not use `ngx.timer(0)`
From the flamegraph, the code in the prometheus plugin does not significantly increase latency, which is most noticeable in the `fix_histogram_bucket_labels` function of nginx-lua-prometheus, even if this function is optimized.
The main delay is due to the regular matching in this function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on pull request #6315: chore: update ngxin-lua-prometheus verison
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on pull request #6315:
URL: https://github.com/apache/apisix/pull/6315#issuecomment-1063746789
> I'm not sure if I modified this file like above. Will the apisix get the new version of the dependency when it starting?
Of course not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on pull request #6315: chore: update ngxin-lua-prometheus verison
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on pull request #6315:
URL: https://github.com/apache/apisix/pull/6315#issuecomment-1068667727
> @tzssangglass ，I have updated the version of nginx-lua-prometheus dependency package. I directly used the new version of dependency package when packaging with dockerfile. My version of apifix is 2.11.0. As shown in the figure. I found that the apifix delay on grafana is not stable, as shown in the figure. At the beginning, it is 190ms, and then it becomes more than 1s.
>
> And, looking at the graph of grafana, the delay is still relatively large. I just piloted a few services. The QPS is less than 100, and the CPU utilization of the gateway is also very low.
>
> 
>
> 
>
> the access.log of the nginx print the variable below: $request_time $upstream_connect_time $upstream_response_time in the access.log , I can not find the record who has so manay delay time.
Building your own image is a black box for us developers (even if you say how you built it, we still don't have access to its details)
Now that the apisix-dev image builds the master branch once a day and pushes it to Docker Hub, can you use this image to verify the problem?
ref: https://hub.docker.com/r/apache/apisix/tags?page=1&name=dev
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander merged pull request #6315: chore: update ngxin-lua-prometheus verison
Posted by GitBox <gi...@apache.org>.
spacewander merged pull request #6315:
URL: https://github.com/apache/apisix/pull/6315
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] jagerzhang commented on pull request #6315: chore: update ngxin-lua-prometheus verison
Posted by GitBox <gi...@apache.org>.
jagerzhang commented on pull request #6315:
URL: https://github.com/apache/apisix/pull/6315#issuecomment-1038843772
mark
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] xianshun163 commented on pull request #6315: chore: update ngxin-lua-prometheus verison
Posted by GitBox <gi...@apache.org>.
xianshun163 commented on pull request #6315:
URL: https://github.com/apache/apisix/pull/6315#issuecomment-1062894374
@tzssangglass I use the apisix 1.11.0, Do you know how can I fix the issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] xianshun163 commented on pull request #6315: chore: update ngxin-lua-prometheus verison
Posted by GitBox <gi...@apache.org>.
xianshun163 commented on pull request #6315:
URL: https://github.com/apache/apisix/pull/6315#issuecomment-1063593287
When the new version was just released, I didn't dare to use it directly. I prefer to fix the bug manually first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] xianshun163 removed a comment on pull request #6315: chore: update ngxin-lua-prometheus verison
Posted by GitBox <gi...@apache.org>.
xianshun163 removed a comment on pull request #6315:
URL: https://github.com/apache/apisix/pull/6315#issuecomment-1063593287
When the new version was just released, I didn't dare to use it directly. I prefer to fix the bug manually first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on pull request #6315: chore: update ngxin-lua-prometheus verison
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on pull request #6315:
URL: https://github.com/apache/apisix/pull/6315#issuecomment-1063590570
> I deploy it like this: this is my docker file: FROM apache/apisix:2.11.0-alpine MAINTAINER xs COPY ./plugins/_.lua /usr/local/apisix/apisix/plugins/ COPY ./discovery/_.lua /usr/local/apisix/apisix/discovery/ COPY ./config/*.yaml /usr/local/apisix/conf/
>
> and then I got my own docker image, I deploy it in the k8s.
Can you wait for the LTS version to be released? Because this PR change is only for the master branch. Rebuilding the image after the next release will affect the dependencies in the image.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] xianshun163 edited a comment on pull request #6315: chore: update ngxin-lua-prometheus verison
Posted by GitBox <gi...@apache.org>.
xianshun163 edited a comment on pull request #6315:
URL: https://github.com/apache/apisix/pull/6315#issuecomment-1063591190
@tzssangglass , I have see the commit, it is only change this file: apisix-master-0.rockspec,
- "nginx-lua-prometheus = 0.20210206",
+ "nginx-lua-prometheus = 0.20220127",
I'm not sure if I modified this file like above. Will the apisix get the new version of the dependency when it starting?
I have a try , It seems not work.
cd /data/apisix-2.11.0/deps/lib/luarocks/rocks-5.1/nginx-lua-prometheus
mv 0.20210206-1 0.20210206-1.bak
cd /data/apisix-2.11.0/
vi rockspec/apisix-master-0.rockspec
- "nginx-lua-prometheus = 0.20210206",
+ "nginx-lua-prometheus = 0.20220127",
./bin/apisix stop
./bin/apisix start
the deps nginx-lua-prometheus can not get the new version. It still use the old version and it can collect the metric.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] xianshun163 edited a comment on pull request #6315: chore: update ngxin-lua-prometheus verison
Posted by GitBox <gi...@apache.org>.
xianshun163 edited a comment on pull request #6315:
URL: https://github.com/apache/apisix/pull/6315#issuecomment-1067908367
@tzssangglass ，I have updated the version of nginx-lua-prometheus dependency package. I directly used the new version of dependency package when packaging with dockerfile.
My version of apifix is 2.11.0.
As shown in the figure. I found that the apifix delay on grafana is not stable, as shown in the figure. At the beginning, it is 190ms, and then it becomes more than 1s.
And, looking at the graph of grafana, the delay is still relatively large.
I just piloted a few services. The QPS is less than 100, and the CPU utilization of the gateway is also very low.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] xianshun163 commented on pull request #6315: chore: update ngxin-lua-prometheus verison
Posted by GitBox <gi...@apache.org>.
xianshun163 commented on pull request #6315:
URL: https://github.com/apache/apisix/pull/6315#issuecomment-1063574118
@tzssangglass sorry , I made a mistake. My version is 2.11.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] xianshun163 edited a comment on pull request #6315: chore: update ngxin-lua-prometheus verison
Posted by GitBox <gi...@apache.org>.
xianshun163 edited a comment on pull request #6315:
URL: https://github.com/apache/apisix/pull/6315#issuecomment-1067908367
@tzssangglass ，I have updated the version of nginx-lua-prometheus dependency package. I directly used the new version of dependency package when packaging with dockerfile.
My version of apifix is 2.11.0.
As shown in the figure. I found that the apifix delay on grafana is not stable, as shown in the figure. At the beginning, it is 190ms, and then it becomes more than 1s.
And, looking at the graph of grafana, the delay is still relatively large.
I just piloted a few services. The QPS is less than 100, and the CPU utilization of the gateway is also very low.
the access.log of the nginx print the variable below:
$request_time $upstream_connect_time $upstream_response_time
in the access.log , I can not find the record who has so manay delay time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] xianshun163 commented on pull request #6315: chore: update ngxin-lua-prometheus verison
Posted by GitBox <gi...@apache.org>.
xianshun163 commented on pull request #6315:
URL: https://github.com/apache/apisix/pull/6315#issuecomment-1068710628
I got it in this issue: #6029
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] xianshun163 commented on pull request #6315: chore: update ngxin-lua-prometheus verison
Posted by GitBox <gi...@apache.org>.
xianshun163 commented on pull request #6315:
URL: https://github.com/apache/apisix/pull/6315#issuecomment-1063578368
I deploy it like this: this is my docker file:
FROM apache/apisix:2.11.0-alpine
MAINTAINER xs
COPY ./plugins/*.lua /usr/local/apisix/apisix/plugins/
COPY ./discovery/*.lua /usr/local/apisix/apisix/discovery/
COPY ./config/*.yaml /usr/local/apisix/conf/
and then I got my own docker image, I deploy it in the k8s.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] xianshun163 commented on pull request #6315: chore: update ngxin-lua-prometheus verison
Posted by GitBox <gi...@apache.org>.
xianshun163 commented on pull request #6315:
URL: https://github.com/apache/apisix/pull/6315#issuecomment-1063591190
@tzssangglass , I have see the commit, it is only change this file: apisix-master-0.rockspec,
- "nginx-lua-prometheus = 0.20210206",
+ "nginx-lua-prometheus = 0.20220127",
I'm not sure if I modified this file like above. Will the apisix get the new version of the dependency when it starting?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] xianshun163 commented on pull request #6315: chore: update ngxin-lua-prometheus verison
Posted by GitBox <gi...@apache.org>.
xianshun163 commented on pull request #6315:
URL: https://github.com/apache/apisix/pull/6315#issuecomment-1067908367
@tzssangglass ，I have updated the version of nginx-lua-prometheus dependency package. I directly used the new version of dependency package when packaging with dockerfile.
My version of apifix is 2.11.0.
As shown in the figure. I found that the apifix delay on grafana is not stable, as shown in the figure. At the beginning, it is 190ms, and then it becomes more than 1s.
And, looking at the graph of grafana, the delay is still relatively large.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] xianshun163 commented on pull request #6315: chore: update ngxin-lua-prometheus verison
Posted by GitBox <gi...@apache.org>.
xianshun163 commented on pull request #6315:
URL: https://github.com/apache/apisix/pull/6315#issuecomment-1067541039
> > I'm not sure if I modified this file like above. Will the apisix get the new version of the dependency when it starting?
>
> Of course not.
I get the dockerfile: https://github.com/apache/apisix-docker/edit/release/apisix-2.11.0/alpine/Dockerfile
modify the deps: "nginx-lua-prometheus = 0.20220127",
and build the images in local.
it is update now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on pull request #6315: chore: update ngxin-lua-prometheus verison
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on pull request #6315:
URL: https://github.com/apache/apisix/pull/6315#issuecomment-1063561017
> @tzssangglass I use the apisix 1.11.0, Do you know how can I fix the issue?
Unfortunately this version is too old, we don't maintain this version, you can consider upgrading, this month we will release a new LTS version -> 2.13.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander merged pull request #6315: chore: update ngxin-lua-prometheus verison
Posted by GitBox <gi...@apache.org>.
spacewander merged pull request #6315:
URL: https://github.com/apache/apisix/pull/6315
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org