You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by "pavankumar-siripurapu (via GitHub)" <gi...@apache.org> on 2024/08/01 07:05:01 UTC
[I] bug: Prometheus plugin consuming more memory. [apisix]
pavankumar-siripurapu opened a new issue, #11456:
URL: https://github.com/apache/apisix/issues/11456
### Current Behavior
This is a follow-up to a [help request](https://github.com/apache/apisix/issues/11442) that was raised a few days ago and discussed with @juzhiyuan. We attempted to resolve the issue by disabling the logging of request bodies in the file-logger plugin, but this did not reduce memory usage. Surprisingly, disabling the Prometheus plugin resulted in a significant reduction in memory usage.
However, without the Prometheus plugin enabled, it becomes challenging for us to track the metrics of APISIX. Therefore, we re-enabled the plugin and encountered the initial issue again. After careful consideration of the extra labels we added, we made some changes and deployed them, which resulted in a memory reduction of 70-80%.
We also want to enable the URI label to obtain metrics based on URI. Additionally, we observed that the memory does not decrease when there is no traffic with the Prometheus plugin enabled. We encountered difficulties implementing the suggested change in the helm configurations mentioned in this link: https://github.com/apache/apisix/blob/release/3.8/conf/config-default.yaml#L175-L177. It would be greatly appreciated if you could provide a sample of how to integrate this change with helm configurations.
### Expected Behavior
_No response_
### Error Logs
_No response_
### Steps to Reproduce
1. Run APISIX with helm chart from api7 on EKS.
2. enable the prometheus plugin with extra labels as below
<img width="420" alt="image" src="https://github.com/user-attachments/assets/3e7d00b8-a3d6-4e72-bb73-cccdc21c2322">
3. Run a performance test with around 3000~4000 requests with different URIs.the
### Environment
- APISIX version (run `apisix version`):
- Operating system (run `uname -a`):
- OpenResty / Nginx version (run `openresty -V` or `nginx -V`):
- etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`):
- APISIX Dashboard version, if relevant:
- Plugin runner version, for issues related to plugin runners:
- LuaRocks version, for installation issues (run `luarocks --version`):
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: Prometheus plugin consuming more memory. [apisix]
Posted by "bzp2010 (via GitHub)" <gi...@apache.org>.
bzp2010 commented on issue #11456:
URL: https://github.com/apache/apisix/issues/11456#issuecomment-2270328881
Hi, @pavankumar-siripurapu.
Although `prometheus` plugin allows you to inject additional labels, it is not feasible to set a field with a high cardinality like uri through a variable.
For example, if you use any wildcard routes or variable routes, they will inflate the uri label indefinitely.
For `apisix_http_status`, which stores things like `"code", "route", "matched_uri", "matched_host", "service", "consumer", "node", "status_code"`, assuming there are 10 possible values for each, and the uri has 100 possible values, this will eventually generate 10*10*10*10*10*10*10*10*100 active series, totaling 10 billion.
This is an immeasurable drain on memory because you can't predict what data your customers will include in the API path. Therefore, 100 possible uri's is just a minimal prediction.
In short, please don't include high cardinality values in labels, or strictly limit its range. Do not do this for global configurations in plugin_attr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: Prometheus plugin consuming more memory. [apisix]
Posted by "juzhiyuan (via GitHub)" <gi...@apache.org>.
juzhiyuan commented on issue #11456:
URL: https://github.com/apache/apisix/issues/11456#issuecomment-2300066000
Hi @pavankumar-siripurapu, as this issue gets resolved, I will close the current issue.
You can create another issue to track other questions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: Prometheus plugin consuming more memory. [apisix]
Posted by "pavankumar-siripurapu (via GitHub)" <gi...@apache.org>.
pavankumar-siripurapu commented on issue #11456:
URL: https://github.com/apache/apisix/issues/11456#issuecomment-2296681988
Thank you, @bzp2010, for providing valuable information. We have successfully eliminated unnecessary labels, resulting in a significant reduction in memory usage. I am now interested in learning how to integrate with a privileged agent. Could you please provide a step-by-step guide for this integration process?
<img width="960" alt="image" src="https://github.com/user-attachments/assets/38a13802-5912-442c-af7f-17875ca6b5ca">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: Prometheus plugin consuming more memory. [apisix]
Posted by "juzhiyuan (via GitHub)" <gi...@apache.org>.
juzhiyuan closed issue #11456: bug: Prometheus plugin consuming more memory.
URL: https://github.com/apache/apisix/issues/11456
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: Prometheus plugin consuming more memory. [apisix]
Posted by "juzhiyuan (via GitHub)" <gi...@apache.org>.
juzhiyuan commented on issue #11456:
URL: https://github.com/apache/apisix/issues/11456#issuecomment-2262312057
Hello @pavankumar-siripurapu, we will follow up and let you know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org