You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by "cataglyphis (via GitHub)" <gi...@apache.org> on 2025/04/02 01:30:33 UTC
[I] bug: apisix prometheus metrics lost, must restart apisix pods [apisix]
cataglyphis opened a new issue, #12112:
URL: https://github.com/apache/apisix/issues/12112
### Current Behavior
- Using Prometheus agent to scrap metrics from apisix pods, and metrics lost about 3 days (no apisix metrics any more)
- After restart apisix pods, everything works well
- This issues occurs after we upgrade apisix from 3.2 to 3.8
### Expected Behavior
_No response_
### Error Logs
_No response_
### Steps to Reproduce
1. Run apisix 3.8.1 in AWS EKS cluster
2. Use Prometheus agent to scrape metrics from apisix pods
### Environment
- APISIX version (run `apisix version`): 3.8.1
- Operating system (run `uname -a`):
- OpenResty / Nginx version (run `openresty -V` or `nginx -V`):
- etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`):
- APISIX Dashboard version, if relevant:
- Plugin runner version, for issues related to plugin runners:
- LuaRocks version, for installation issues (run `luarocks --version`):
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: apisix prometheus metrics lost, must restart apisix pods [apisix]
Posted by "Baoyuantop (via GitHub)" <gi...@apache.org>.
Baoyuantop commented on issue #12112:
URL: https://github.com/apache/apisix/issues/12112#issuecomment-2771413689
From the error logs, these two errors are related to the `http_latency` and `bandwidth` metrics respectively, indicating that these metrics generate a large number of key values under a large number of routes and nodes, resulting in insufficient space in shared memory.
You can try increasing the size of the prometheus-metrics shared dictionary, or using the metrics' expire configuration to reduce the number of metrics stored for the long term, and restarting APISIX to take effect
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: apisix prometheus metrics lost, must restart apisix pods [apisix]
Posted by "cataglyphis (via GitHub)" <gi...@apache.org>.
cataglyphis commented on issue #12112:
URL: https://github.com/apache/apisix/issues/12112#issuecomment-2774667420
After reviewed the helm chart of Apisix, there is no way to set shared dict space for `prometheus-metrics` in values, we must add below config in apisix configmap manually.
```yaml
nginx_config:
...
http:
...
lua_shared_dict:
prometheus-metrics: 100m
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: apisix prometheus metrics lost, must restart apisix pods [apisix]
Posted by "cataglyphis (via GitHub)" <gi...@apache.org>.
cataglyphis commented on issue #12112:
URL: https://github.com/apache/apisix/issues/12112#issuecomment-2771286690
is this error logs related?
```
2025/04/01 20:09:48 [error] 74#74: *127311366 [lua] prometheus_resty_counter.lua:39: increasing counter in shdict: lru eviction: key=http_latency_bucket{type="request",route="env-prod-api_api",service="",consumer="",node="10.170.104.3",le="0050.0"}, context: ngx.timer
2025/04/01 20:09:48 [error] 74#74: *127311366 [lua] prometheus_resty_counter.lua:39: increasing counter in shdict: lru eviction: key=bandwidth{type="egress",route="ing_xxx_mall-portal_7202e927",service="",consumer="",node="10.170.159.243"}, context: ngx.timer
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: apisix prometheus metrics lost, must restart apisix pods [apisix]
Posted by "cataglyphis (via GitHub)" <gi...@apache.org>.
cataglyphis commented on issue #12112:
URL: https://github.com/apache/apisix/issues/12112#issuecomment-2772626451
> From the error logs, these two errors are related to the `http_latency` and `bandwidth` metrics respectively, indicating that these metrics generate a large number of key values under a large number of routes and nodes, resulting in insufficient space in shared memory.
>
> You can try increasing the size of the prometheus-metrics shared dictionary, or using the metrics' expire configuration to reduce the number of metrics stored for the long term, and restarting APISIX to take effect
I have another question: the value of apisix_shared_dict_free_space_bytes for Prometheus metrics is currently almost 0, while the default capacity is set to 10 MiB. What is the recommended value for the shared dictionary space for Prometheus?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: apisix prometheus metrics lost, must restart apisix pods [apisix]
Posted by "cataglyphis (via GitHub)" <gi...@apache.org>.
cataglyphis commented on issue #12112:
URL: https://github.com/apache/apisix/issues/12112#issuecomment-2772538098
> From the error logs, these two errors are related to the `http_latency` and `bandwidth` metrics respectively, indicating that these metrics generate a large number of key values under a large number of routes and nodes, resulting in insufficient space in shared memory.
>
> You can try increasing the size of the prometheus-metrics shared dictionary, or using the metrics' expire configuration to reduce the number of metrics stored for the long term, and restarting APISIX to take effect
`the metrics' expire configuration`: apisix 3.8 has this configuration yet?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: apisix prometheus metrics lost, must restart apisix pods [apisix]
Posted by "cataglyphis (via GitHub)" <gi...@apache.org>.
cataglyphis closed issue #12112: bug: apisix prometheus metrics lost, must restart apisix pods
URL: https://github.com/apache/apisix/issues/12112
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: apisix prometheus metrics lost, must restart apisix pods [apisix]
Posted by "Baoyuantop (via GitHub)" <gi...@apache.org>.
Baoyuantop commented on issue #12112:
URL: https://github.com/apache/apisix/issues/12112#issuecomment-2771113526
Hi @cataglyphis, are there any error logs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: apisix prometheus metrics lost, must restart apisix pods [apisix]
Posted by "Baoyuantop (via GitHub)" <gi...@apache.org>.
Baoyuantop commented on issue #12112:
URL: https://github.com/apache/apisix/issues/12112#issuecomment-2775112118
> [@Baoyuantop](https://github.com/Baoyuantop) I have another question: the value of `apisix_shared_dict_free_space_bytes` for Prometheus metrics is currently almost 0, while the default capacity is set to 10 MiB. What is the recommended value for the shared dictionary space for Prometheus?
This needs to be set according to the number of requests you have, see the https://github.com/apache/apisix/issues/11948#issuecomment-2676366094
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org