You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2022/06/08 10:32:09 UTC
[GitHub] [apisix] crazyMonkey1995 opened a new issue, #7211: bug: prometheus metrics request affect normal business requests
crazyMonkey1995 opened a new issue, #7211:
URL: https://github.com/apache/apisix/issues/7211
### Current Behavior
At present, we have enabled the prometheus plugin. Since we have more routes (4000+), the size of the lua share dict of prometheus is 512M (larger). When the amount of data becomes larger, prometheus.export_metrics() will cause the cpu-usage of one worker process very hight, and requests on this worker process will be greatly affected.
Maybe there is some way to separate the worker process of internal server (such as admin and prometheus) and the actual business server?
### Expected Behavior
-
### Error Logs
-
### Steps to Reproduce
1. enable prometheus plugin
2. maybe set lua share dict of prometheus to 512m
3. increase prometheus data(by send many request?)
4. curl the metric url
### Environment
- APISIX version (run `apisix version`):
- Operating system (run `uname -a`):
- OpenResty / Nginx version (run `openresty -V` or `nginx -V`):
- etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`):
- APISIX Dashboard version, if relevant:
- Plugin runner version, for issues related to plugin runners:
- LuaRocks version, for installation issues (run `luarocks --version`):
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander closed issue #7211: bug: prometheus metrics cause high cpu usage
Posted by GitBox <gi...@apache.org>.
spacewander closed issue #7211: bug: prometheus metrics cause high cpu usage
URL: https://github.com/apache/apisix/issues/7211
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] jagerzhang commented on issue #7211: bug: prometheus metrics cause high cpu usage
Posted by GitBox <gi...@apache.org>.
jagerzhang commented on issue #7211:
URL: https://github.com/apache/apisix/issues/7211#issuecomment-1150830693
> > Maybe there is some way to separate the worker process of internal server (such as admin and prometheus) and the actual business server?
>
> We have already done it in the Enterprise version and received good feedback. @membphis Is it possible to open source the change?
Can this plan also solve the previous problem：https://github.com/apache/apisix/issues/5755
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #7211: bug: prometheus metrics cause high cpu usage
Posted by GitBox <gi...@apache.org>.
spacewander commented on issue #7211:
URL: https://github.com/apache/apisix/issues/7211#issuecomment-1189706469
Let's discuss it at https://github.com/apache/apisix/issues/7211.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] xuminwlt commented on issue #7211: bug: prometheus metrics cause high cpu usage
Posted by GitBox <gi...@apache.org>.
xuminwlt commented on issue #7211:
URL: https://github.com/apache/apisix/issues/7211#issuecomment-1165645546
Same question to me. Is there any way we can do , not close it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] jagerzhang commented on issue #7211: bug: prometheus metrics cause high cpu usage
Posted by GitBox <gi...@apache.org>.
jagerzhang commented on issue #7211:
URL: https://github.com/apache/apisix/issues/7211#issuecomment-1184157595
> > Same question to me. Is there any way we can do , not close it.
>
> I have seen that among the metrics in Prometheus, the amount of delay metrics data is very large. This metrics is not very important to me, so I removed it in exporter.lua and mount by configmap. Now the memory is greatly reduced, and the CPU usage during collection is also greatly reduced. Looking forward to better solutions.
@crazyMonkey1995
Looking forward to your best practices
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on issue #7211: bug: prometheus metrics cause high cpu usage
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #7211:
URL: https://github.com/apache/apisix/issues/7211#issuecomment-1150565340
Do you mean the method `prometheus:metric_data`? https://github.com/api7/lua-resty-prometheus/blob/master/prometheus.lua#L775
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] membphis commented on issue #7211: bug: prometheus metrics cause high cpu usage
Posted by GitBox <gi...@apache.org>.
membphis commented on issue #7211:
URL: https://github.com/apache/apisix/issues/7211#issuecomment-1150873084
> We have already done it in the Enterprise version and received good feedback.
> @membphis
> Is it possible to open source the change?
we do not have this plan now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] jinjianming commented on issue #7211: bug: prometheus metrics cause high cpu usage
Posted by "jinjianming (via GitHub)" <gi...@apache.org>.
jinjianming commented on issue #7211:
URL: https://github.com/apache/apisix/issues/7211#issuecomment-1625074973
> > 对我同样的问题。有什么办法可以做到，而不是关闭它。
>
> 我看到Prometheus中的指标中，延迟指标数据量非常大。这个指标对我来说不是很重要，所以我在exporter.lua中删除了它并通过configmap挂载。现在内存大大减少了，收集时的CPU占用率也大大减少了。期待更好的解决方案。
Seeing that the author has removed some information and reduced some latency based on actual business needs, can you share some code examples on how to remove Apisix_ HTTP_ Latency_ I am extremely grateful for encountering the same problem with indicators such as buckets;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] crazyMonkey1995 commented on issue #7211: bug: prometheus metrics cause high cpu usage
Posted by GitBox <gi...@apache.org>.
crazyMonkey1995 commented on issue #7211:
URL: https://github.com/apache/apisix/issues/7211#issuecomment-1165666309
> Same question to me. Is there any way we can do , not close it.
I have seen that among the metrics in Prometheus, the amount of delay metrics data is very large. This metrics is not very important to me, so I removed it in exporter.lua and mount by configmap. Now the memory is greatly reduced, and the CPU usage during collection is also greatly reduced. Looking forward to better solutions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] Dieken commented on issue #7211: bug: prometheus metrics cause high cpu usage
Posted by GitBox <gi...@apache.org>.
Dieken commented on issue #7211:
URL: https://github.com/apache/apisix/issues/7211#issuecomment-1171281836
> > We have already done it in the Enterprise version and received good feedback.
> > @membphis
> > Is it possible to open source the change?
>
> we do not have this plan now.
@membphis "PMC member of Apache APISIX, Co-founder and CTO of API7.ai , we are hiring engineers, mail me."
这个身份，还 hiring，你们这么做开源项目是不是形象不太合适，新功能藏在企业版可以理解，明知道有问题修正也不放出来，你们还是闭源了好好做商业吧，形象和钱都可以有了。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] crazyMonkey1995 commented on issue #7211: bug: prometheus metrics cause high cpu usage
Posted by GitBox <gi...@apache.org>.
crazyMonkey1995 commented on issue #7211:
URL: https://github.com/apache/apisix/issues/7211#issuecomment-1150574227
> Do you mean the method `prometheus:metric_data`? https://github.com/api7/lua-resty-prometheus/blob/master/prometheus.lua#L775
Enable prometheus plugin and access this location⬆️
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] crazyMonkey1995 commented on issue #7211: bug: prometheus metrics cause high cpu usage
Posted by GitBox <gi...@apache.org>.
crazyMonkey1995 commented on issue #7211:
URL: https://github.com/apache/apisix/issues/7211#issuecomment-1150598627
notice that codes changed about prometheus, my apisix version: 2.14.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #7211: bug: prometheus metrics cause high cpu usage
Posted by GitBox <gi...@apache.org>.
spacewander commented on issue #7211:
URL: https://github.com/apache/apisix/issues/7211#issuecomment-1150714437
> Maybe there is some way to separate the worker process of internal server (such as admin and prometheus) and the actual business server?
We have already done it in the Enterprise version and received good feedback.
@membphis
Is it possible to open source the change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #7211: bug: prometheus metrics cause high cpu usage
Posted by GitBox <gi...@apache.org>.
spacewander commented on issue #7211:
URL: https://github.com/apache/apisix/issues/7211#issuecomment-1166520623
> > > Same question to me. Is there any way we can do , not close it.
> >
> >
> > I have seen that among the metrics in Prometheus, the amount of delay metrics data is very large. This metrics is not very important to me, so I removed it in exporter.lua and mount by configmap. Now the memory is greatly reduced, and the CPU usage during collection is also greatly reduced. Looking forward to better solutions.
>
> Or can we add some configuration items of the Prometheus plugin to support selecting what data to collect
Similar to https://github.com/apache/apisix/issues/4273?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] crazyMonkey1995 commented on issue #7211: bug: prometheus metrics cause high cpu usage
Posted by GitBox <gi...@apache.org>.
crazyMonkey1995 commented on issue #7211:
URL: https://github.com/apache/apisix/issues/7211#issuecomment-1165669868
> > Same question to me. Is there any way we can do , not close it.
>
> I have seen that among the metrics in Prometheus, the amount of delay metrics data is very large. This metrics is not very important to me, so I removed it in exporter.lua and mount by configmap. Now the memory is greatly reduced, and the CPU usage during collection is also greatly reduced. Looking forward to better solutions.
Or can we add some configuration items of the Prometheus plugin to support selecting what data to collect
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org