You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2022/06/29 04:26:47 UTC
[GitHub] [apisix] huayi550 opened a new issue, #7349: bug: prometheus throw no memory while logging request when there are too many upstream and routes
huayi550 opened a new issue, #7349:
URL: https://github.com/apache/apisix/issues/7349
### Current Behavior
On the grafanal panel，we found error for nginx metrics such as below
so we check the prometheus target of one APISIX instance
`http://10.xx.xx.xx:9091/apisix/prometheus/metrics`
```
# HELP apisix_nginx_http_current_connections Number of HTTP connections
# TYPE apisix_nginx_http_current_connections gauge
apisix_nginx_http_current_connections{state="accepted"} 5011060
apisix_nginx_http_current_connections{state="active"} 6
apisix_nginx_http_current_connections{state="handled"} 5011060
apisix_nginx_http_current_connections{state="reading"} 0
apisix_nginx_http_current_connections{state="waiting"} 1
apisix_nginx_http_current_connections{state="writing"} 5
# HELP apisix_nginx_metric_errors_total Number of nginx-lua-prometheus errors
# TYPE apisix_nginx_metric_errors_total counter
apisix_nginx_metric_errors_total 1372
# HELP apisix_node_info Info of APISIX node
# TYPE apisix_node_info gauge
apisix_node_info{hostname="ecs-apisix-0001"} 1
```
- Usage on production environment
| APISIX Instance | Etcd Cluster Instance | Upstream | Routes|
|-- | -- | -- | --|
| 3 | 3 | 28 | 614|
<!--EndFragment-->
</body>
</html>
Then we check the error on /apisix/logs/error.log ，found the logs about prometheus below：
### Expected Behavior
_No response_
### Error Logs
```
2022/06/29 10:51:16 [error] 32404#32404: *92610688 [lua] prometheus.lua:872: log_error(): Unexpected error adding a key: no memory while logging request, client: 10.11.0.43, server: _, request: "POST /parkMap/modifyMapName HTTP/1.0", upstream: "http://96.11.0.4:8889/parkMap/modifyMapName", host: "apigw.*****.com.cn"
2022/06/29 10:51:16 [error] 32404#32404: *92610688 [lua] prometheus.lua:872: log_error(): Unexpected error adding a key: no memory while logging request, client: 10.11.0.43, server: _, request: "POST /parkMap/modifyMapName HTTP/1.0", upstream: "http://96.11.0.4:8889/parkMap/modifyMapName", host: "apigw.*****.com.cn"
2022/06/29 10:51:16 [error] 32404#32404: *92610688 [lua] prometheus.lua:872: log_error(): Unexpected error adding a key: no memory while logging request, client: 10.11.0.43, server: _, request: "POST /parkMap/modifyMapName HTTP/1.0", upstream: "http://96.11.0.4:8889/parkMap/modifyMapName", host: "apigw.*****.com.cn"
```
### Steps to Reproduce
we have enabled the global plugin : `prometheus` ，the config is below
```
{
"disable": false,
"prefer_name": false
}
```
### Environment
- APISIX version (run `apisix version`): 2.13.1
- Operating system (run `uname -a`): CentOS7.9
- OpenResty / Nginx version (run `openresty -V` or `nginx -V`): openresty/1.19.9.1
- etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`): 3.5.2
- APISIX Dashboard version, if relevant: 2.13.0
- Plugin runner version, for issues related to plugin runners:
- LuaRocks version, for installation issues (run `luarocks --version`):
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] huayi550 commented on issue #7349: bug: prometheus throw no memory while logging request when there are too many upstream and routes
Posted by GitBox <gi...@apache.org>.
huayi550 commented on issue #7349:
URL: https://github.com/apache/apisix/issues/7349#issuecomment-1169867234
I have tried this，it''s ok.
thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #7349: bug: prometheus throw no memory while logging request when there are too many upstream and routes
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #7349:
URL: https://github.com/apache/apisix/issues/7349#issuecomment-1169841842
try to update `prometheus-metrics` in `config.yaml`, example:
```
apisix:
http:
lua_shared_dict:
prometheus-metrics: 10m
```
and you can read the relevant issue: https://github.com/apache/apisix/search?q=no+memory+while+logging+request+is%3Aissue&type=issues to know more about possible problems caused by increasing prometheus memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on issue #7349: bug: prometheus throw no memory while logging request when there are too many upstream and routes
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #7349:
URL: https://github.com/apache/apisix/issues/7349#issuecomment-1169739229
@huayi550 How many memory do you allocate for Prometheus metrics in APISIX.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] huayi550 closed issue #7349: bug: prometheus throw no memory while logging request when there are too many upstream and routes
Posted by GitBox <gi...@apache.org>.
huayi550 closed issue #7349: bug: prometheus throw no memory while logging request when there are too many upstream and routes
URL: https://github.com/apache/apisix/issues/7349
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] huayi550 commented on issue #7349: bug: prometheus throw no memory while logging request when there are too many upstream and routes
Posted by GitBox <gi...@apache.org>.
huayi550 commented on issue #7349:
URL: https://github.com/apache/apisix/issues/7349#issuecomment-1169520770
Are we using it incorrectly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org