You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2021/09/18 03:04:00 UTC
[GitHub] [apisix] suninuni opened a new issue #5092: memory leak when enable prometheus as global
suninuni opened a new issue #5092:
URL: https://github.com/apache/apisix/issues/5092
### Issue description
When the prometheus global plugin is enabled, the memory of apisix continues to grow abnormally.
### Environment
- apisix version (cmd: `apisix version`): 2.7
### Steps to reproduce
- deploy httpbin service in k8s ([step in apisix-inrgess-controller](https://apisix.apache.org/docs/ingress-controller/practices/proxy-the-httpbin-service))
- enable prometheus as global rule and set perfer_name as true (lua_shared_dict.prometheus-metrics is 10m)
- mock http request throw ab with high concurrency `ab -n 2000000 -c 2000 http://xxxxx`
### Actual result
- memory keep growing

### Error log
- `*45077733 [lua] prometheus.lua:860: log_error(): Unexpected error adding a key: no memory while logging request,`
### Expected result
- The memory should not continue to grow.
- Prometheus should release a part of the memory allocated to it after it is used up to avoid continuous errors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] suninuni commented on issue #5092: memory leak when enable prometheus as global
Posted by GitBox <gi...@apache.org>.
suninuni commented on issue #5092:
URL: https://github.com/apache/apisix/issues/5092#issuecomment-1006207488
> > So it seems that this problem no longer exists with APISIX `2.11`, I will continue to observe. Thanks for your support!
>
> do you update your apisix to `2.11`?
yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] suninuni commented on issue #5092: memory leak when enable prometheus as global
Posted by GitBox <gi...@apache.org>.
suninuni commented on issue #5092:
URL: https://github.com/apache/apisix/issues/5092#issuecomment-1002848302
> @suninuni the latest version of APISIX is 2.11 now, can you make a try with APISIX `2.11`? We can not sure if we have fixed this issue.
>
> If you still have this problem, please let us know.
The same error log as before.
```
2021/12/30 02:36:43 [error] 50#50: *20414229 [lua] prometheus.lua:860: log_error(): Unexpected error adding a key: no memory while logging request
```
And now I try to set `lua_shared_dict.prometheus-metrics` as 100m, FYI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] suninuni commented on issue #5092: memory leak when enable prometheus as global
Posted by GitBox <gi...@apache.org>.
suninuni commented on issue #5092:
URL: https://github.com/apache/apisix/issues/5092#issuecomment-1002888234
> And now I try to set lua_shared_dict.prometheus-metrics as 100m
It seems that the memory does not increase after it has increased to a certain value, and I have not found `no memory while logging request` log any more.

So it seems that this problem no longer exists with APISIX `2.11`, I will continue to observe. Thanks for your support!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] suninuni commented on issue #5092: memory leak when enable prometheus as global
Posted by GitBox <gi...@apache.org>.
suninuni commented on issue #5092:
URL: https://github.com/apache/apisix/issues/5092#issuecomment-1005496987
@tokers @membphis FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] suninuni edited a comment on issue #5092: memory leak when enable prometheus as global
Posted by GitBox <gi...@apache.org>.
suninuni edited a comment on issue #5092:
URL: https://github.com/apache/apisix/issues/5092#issuecomment-1004420968
> > Unexpected error adding a key
>
> So the memory increasing is normal and the no memory issue is just because the pre-defined lua shared dict too small?
After a period of stability, memory has being continue increasing until oom...

But there is no nginx metrics errors like `Unexpected error adding a key`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] suninuni edited a comment on issue #5092: memory leak when enable prometheus as global
Posted by GitBox <gi...@apache.org>.
suninuni edited a comment on issue #5092:
URL: https://github.com/apache/apisix/issues/5092#issuecomment-922251541
> The configuration of lua shdict is not available until 2.8, so you may need to modify the ngx_tpl.lua
We are now deploying apisix in the test environment, so I will upgrade apisix to 2.9 and retest in next week.
> How many metrics/labels are there in the Prometheus metrics?
I counted the number of metrics rows in two pods with different memory consumption. Is this magnitude acceptable?
memory 763m: 18869

memory 596m: 17924

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] suninuni commented on issue #5092: memory leak when enable prometheus as global
Posted by GitBox <gi...@apache.org>.
suninuni commented on issue #5092:
URL: https://github.com/apache/apisix/issues/5092#issuecomment-1006230384

Some monitoring datas which may be helpful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #5092: memory leak when enable prometheus as global
Posted by GitBox <gi...@apache.org>.
spacewander commented on issue #5092:
URL: https://github.com/apache/apisix/issues/5092#issuecomment-922242153
First of all, we need to check if the error log is caused by the memory used by Prometheus is unbound or just because the default size is too small.
The configuration of lua shdict is not available until 2.8, so you may need to modify the ngx_tpl.lua: https://github.com/apache/apisix/pull/4524
If the Prometheus memory usage grows without limit, it will consume all the memory configurated. Otherwise, the memory usage stops growing in a level.
We also need to compare the Prometheus metrics before/after the http request. Prometheus client is a well-known memory consumer. How many metrics/labels are there in the Prometheus metrics?
You can also use X-Ray to diagnose the memory issue: https://openresty.com.cn/cn/xray/. Note that I am not the developer of X-Ray (it is a commercial product developed by others).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] suninuni commented on issue #5092: memory leak when enable prometheus as global
Posted by GitBox <gi...@apache.org>.
suninuni commented on issue #5092:
URL: https://github.com/apache/apisix/issues/5092#issuecomment-1004420968
> > Unexpected error adding a key
>
> So the memory increasing is normal and the no memory issue is just because the pre-defined lua shared dict too small?
After a period of stability, memory increased until oom...

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] membphis commented on issue #5092: memory leak when enable prometheus as global
Posted by GitBox <gi...@apache.org>.
membphis commented on issue #5092:
URL: https://github.com/apache/apisix/issues/5092#issuecomment-1005560937
> So it seems that this problem no longer exists with APISIX `2.11`, I will continue to observe. Thanks for your support!
do you update your apisix to `2.11`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] suninuni edited a comment on issue #5092: memory leak when enable prometheus as global
Posted by GitBox <gi...@apache.org>.
suninuni edited a comment on issue #5092:
URL: https://github.com/apache/apisix/issues/5092#issuecomment-1006207488
> > So it seems that this problem no longer exists with APISIX `2.11`, I will continue to observe. Thanks for your support!
>
> do you update your apisix to `2.11`?
yes

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] suninuni commented on issue #5092: memory leak when enable prometheus as global
Posted by GitBox <gi...@apache.org>.
suninuni commented on issue #5092:
URL: https://github.com/apache/apisix/issues/5092#issuecomment-922251541
> The configuration of lua shdict is not available until 2.8, so you may need to modify the ngx_tpl.lua
We are now deploying apisix in the test environment, so I will upgrade apisix to 2.9 and retest in next week.
> How many metrics/labels are there in the Prometheus metrics?
I counted the number of metrics rows in two pods with different memory consumption. Is this magnitude acceptable?
memory 763m: 18869

memory 596m: 17924

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] membphis commented on issue #5092: memory leak when enable prometheus as global
Posted by GitBox <gi...@apache.org>.
membphis commented on issue #5092:
URL: https://github.com/apache/apisix/issues/5092#issuecomment-1000182799
@suninuni the latest version of APISIX is 2.11 now, can you make a try with APISIX `2.11`? We can not sure if we have fixed this issue.
If you still have this problem, please let us know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] suninuni commented on issue #5092: memory leak when enable prometheus as global
Posted by GitBox <gi...@apache.org>.
suninuni commented on issue #5092:
URL: https://github.com/apache/apisix/issues/5092#issuecomment-922168339
releated https://github.com/apache/apisix/issues/3917#issuecomment-921736339
@tzssangglass FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #5092: memory leak when enable prometheus as global
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #5092:
URL: https://github.com/apache/apisix/issues/5092#issuecomment-922487651
IMO, this is not normal, but it still depends on the size of the lua share dict assigned to prometheus in your runtime nginx.conf, and the number of concurrent requests. I suggest that you try with X-Ray that @spacewander mentioned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] suninuni commented on issue #5092: memory leak when enable prometheus as global
Posted by GitBox <gi...@apache.org>.
suninuni commented on issue #5092:
URL: https://github.com/apache/apisix/issues/5092#issuecomment-922234792

It's keep growing, FYI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on issue #5092: memory leak when enable prometheus as global
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #5092:
URL: https://github.com/apache/apisix/issues/5092#issuecomment-1002944543
> Unexpected error adding a key
So the memory increasing is normal and the no memory issue is just because the pre-defined lua shared dict too small?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] suninuni edited a comment on issue #5092: memory leak when enable prometheus as global
Posted by GitBox <gi...@apache.org>.
suninuni edited a comment on issue #5092:
URL: https://github.com/apache/apisix/issues/5092#issuecomment-1002848302
> @suninuni the latest version of APISIX is 2.11 now, can you make a try with APISIX `2.11`? We can not sure if we have fixed this issue.
>
> If you still have this problem, please let us know.
The same error log as before.
```
2021/12/30 02:36:43 [error] 50#50: *20414229 [lua] prometheus.lua:860: log_error(): Unexpected error adding a key: no memory while logging request
```
And now I try to set `lua_shared_dict.prometheus-metrics` as 100m, FYI.
Although not mentioned before, I also set up dict before upgrading and also encountered the memory leak problem, FYI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] suninuni edited a comment on issue #5092: memory leak when enable prometheus as global
Posted by GitBox <gi...@apache.org>.
suninuni edited a comment on issue #5092:
URL: https://github.com/apache/apisix/issues/5092#issuecomment-1004420968
> > Unexpected error adding a key
>
> So the memory increasing is normal and the no memory issue is just because the pre-defined lua shared dict too small?
After a period of stability, memory increasing until oom...

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org