You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2021/03/26 06:44:06 UTC
[GitHub] [apisix] diaosj opened a new issue #3917: request help: question about Prometheus memory consumption
diaosj opened a new issue #3917:
URL: https://github.com/apache/apisix/issues/3917
### Issue description
I noticed this message in `logs/error.log`:
```
2021/03/26 14:37:26 [error] 38#38: *130656606 [lua] prometheus.lua:732: log_error(): Unexpected error adding a key: no memory while logging request
```
I guess the memory for Prometheus metrics has an upbound limit. What is that value? I may need to config more memory for APISIX.
### Environment
* apisix version (cmd: `apisix version`): 2.1
* OS (cmd: `uname -a`): Linux
* OpenResty / Nginx version (cmd: `nginx -V` or `openresty -V`): openresty/1.19.3.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] Yiyiyimu commented on issue #3917: request help: question about Prometheus memory consumption
Posted by GitBox <gi...@apache.org>.
Yiyiyimu commented on issue #3917:
URL: https://github.com/apache/apisix/issues/3917#issuecomment-873236843
Hi @Applenice I think it would import no breaking changes so it's good to go~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #3917: request help: question about Prometheus memory consumption
Posted by GitBox <gi...@apache.org>.
spacewander commented on issue #3917:
URL: https://github.com/apache/apisix/issues/3917#issuecomment-874090079
What about configuring with the original dash style?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] suninuni commented on issue #3917: request help: question about Prometheus memory consumption
Posted by GitBox <gi...@apache.org>.
suninuni commented on issue #3917:
URL: https://github.com/apache/apisix/issues/3917#issuecomment-920704670
When I set prometheus-metrics as 100m in configmap apisix, apisix throw an error about key conflicted.
```ynl
http:
lua_shared_dicts: # add custom shared cache to nginx.conf
prometheus-metrics: 100m
```
```shell
/usr/local/openresty/luajit/bin/luajit ./apisix/cli/apisix.lua init
WARNING: using fixed Admin API token has security risk.
Please modify "admin_key" in conf/config.yaml .
/usr/local/openresty/luajit/bin/luajit ./apisix/cli/apisix.lua init_etcd
2021/09/16 08:28:56 [emerg] 1#1: the size 104857600 of shared memory zone "prometheus-metrics" conflicts with already declared size 10485760 in /usr/local/apisix/conf/nginx.conf:64
nginx: [emerg] the size 104857600 of shared memory zone "prometheus-metrics" conflicts with already declared size 10485760 in /usr/local/apisix/conf/nginx.conf:64
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] Applenice commented on issue #3917: request help: question about Prometheus memory consumption
Posted by GitBox <gi...@apache.org>.
Applenice commented on issue #3917:
URL: https://github.com/apache/apisix/issues/3917#issuecomment-873762611
> @Applenice Have you tried this:
>
> ```yaml
> lua_shared_dict internal_status {* http.lua_shared_dict["plugin-limit-req"] *};
> ```
I just tested this way of writing and it does work. I learned.
So do I need to change the PR content? Would it be better to keep one style?😃 Personally, I think it's a bit strange to have two ways of handling the same feature because of different naming formats. Of course my idea may not be the best.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander closed issue #3917: request help: question about Prometheus memory consumption
Posted by GitBox <gi...@apache.org>.
spacewander closed issue #3917:
URL: https://github.com/apache/apisix/issues/3917
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] suninuni edited a comment on issue #3917: request help: question about Prometheus memory consumption
Posted by GitBox <gi...@apache.org>.
suninuni edited a comment on issue #3917:
URL: https://github.com/apache/apisix/issues/3917#issuecomment-921549517
And another question is can this completely solve the problem? In my opinion, expanding this value can only alleviate this problem, not solve it. If the memory is full, shouldn't it release part of it?
In my test, this error will continue to appear after the memory is full, and it seems that no request can continue to be recorded. And the memory of the apisix pod will continue to grow, even to 900m. Although I am not sure if it is caused by this problem, it seems that there is something wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] diaosj commented on issue #3917: request help: question about Prometheus memory consumption
Posted by GitBox <gi...@apache.org>.
diaosj commented on issue #3917:
URL: https://github.com/apache/apisix/issues/3917#issuecomment-808949145
> https://github.com/apache/apisix/blob/83931ba0a5c95e04ec867a4ffd0821d426ca0d27/apisix/cli/ngx_tpl.lua#L131
>
> I would appreciate it if you can make it configurable.
Thanks!
Looks like I can config it in `config-default.yaml`. I will try this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] suninuni commented on issue #3917: request help: question about Prometheus memory consumption
Posted by GitBox <gi...@apache.org>.
suninuni commented on issue #3917:
URL: https://github.com/apache/apisix/issues/3917#issuecomment-921542527
> try
>
> ```
> apisix:
> http:
> lua_shared_dict:
> prometheus-metrics: 10m
> ```
>
> in `config.yaml`
I set this to 100m both in `apisix.http.lua_shared_dict.prometheus-metrics` AND `nginx_config.http.lua_shared_dict.prometheus-metrics`, but value in conf/nginx.conf is still be 10m.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #3917: request help: question about Prometheus memory consumption
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #3917:
URL: https://github.com/apache/apisix/issues/3917#issuecomment-921519816
try
```
apisix:
http:
lua_shared_dict:
prometheus-metrics: 10m
```
in `config.yaml`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] Applenice commented on issue #3917: request help: question about Prometheus memory consumption
Posted by GitBox <gi...@apache.org>.
Applenice commented on issue #3917:
URL: https://github.com/apache/apisix/issues/3917#issuecomment-874130176
> What about configuring with the original dash style?
Agree, so that it can be consistent with the naming of the plugin.😊
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #3917: request help: question about Prometheus memory consumption
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #3917:
URL: https://github.com/apache/apisix/issues/3917#issuecomment-920947657
> When I set prometheus-metrics as 100m in configmap apisix, apisix throw an error about key conflicted.
>
> Apisix version is 2.7
>
> ```
> http:
> lua_shared_dicts: # add custom shared cache to nginx.conf
> prometheus-metrics: 100m
> ```
>
> ```shell
> /usr/local/openresty/luajit/bin/luajit ./apisix/cli/apisix.lua init
> WARNING: using fixed Admin API token has security risk.
> Please modify "admin_key" in conf/config.yaml .
> /usr/local/openresty/luajit/bin/luajit ./apisix/cli/apisix.lua init_etcd
> 2021/09/16 08:28:56 [emerg] 1#1: the size 104857600 of shared memory zone "prometheus-metrics" conflicts with already declared size 10485760 in /usr/local/apisix/conf/nginx.conf:64
> nginx: [emerg] the size 104857600 of shared memory zone "prometheus-metrics" conflicts with already declared size 10485760 in /usr/local/apisix/conf/nginx.conf:64
> ```
This has nothing to do with 100m, the error logs have clearly indicated this to you.
the shared dict `prometheus-metrics` has been already defined, see: https://github.com/apache/apisix/blob/d6c09fba6543c9fc28520ef7172596a01417222f/apisix/cli/ngx_tpl.lua#L171
so you can not define shared dict with the same name(`prometheus-metrics`) in a custom configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] suninuni commented on issue #3917: request help: question about Prometheus memory consumption
Posted by GitBox <gi...@apache.org>.
suninuni commented on issue #3917:
URL: https://github.com/apache/apisix/issues/3917#issuecomment-921549517
And another question is can this completely solve the problem? In my opinion, expanding this value can only alleviate this problem, not solve it. If the memory is full, shouldn't it release part of it?
In my test, this error will continue to appear after the memory is full, and it seems that no request can continue to be recorded. And the memory of the apisix pod will continue to grow, even to 900m. Although I am not sure if it is caused by this problem, it seems that there is something wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] Applenice commented on issue #3917: request help: question about Prometheus memory consumption
Posted by GitBox <gi...@apache.org>.
Applenice commented on issue #3917:
URL: https://github.com/apache/apisix/issues/3917#issuecomment-873306583
> Hi @Applenice I think it would import no breaking changes so it's good to go~
Okay, I'll fix it.😃
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] suninuni commented on issue #3917: request help: question about Prometheus memory consumption
Posted by GitBox <gi...@apache.org>.
suninuni commented on issue #3917:
URL: https://github.com/apache/apisix/issues/3917#issuecomment-921377180
> > When I set prometheus-metrics as 100m in configmap apisix, apisix throw an error about key conflicted.
> > Apisix version is 2.7
> > ```
> > http:
> > lua_shared_dicts: # add custom shared cache to nginx.conf
> > prometheus-metrics: 100m
> > ```
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > ```shell
> > /usr/local/openresty/luajit/bin/luajit ./apisix/cli/apisix.lua init
> > WARNING: using fixed Admin API token has security risk.
> > Please modify "admin_key" in conf/config.yaml .
> > /usr/local/openresty/luajit/bin/luajit ./apisix/cli/apisix.lua init_etcd
> > 2021/09/16 08:28:56 [emerg] 1#1: the size 104857600 of shared memory zone "prometheus-metrics" conflicts with already declared size 10485760 in /usr/local/apisix/conf/nginx.conf:64
> > nginx: [emerg] the size 104857600 of shared memory zone "prometheus-metrics" conflicts with already declared size 10485760 in /usr/local/apisix/conf/nginx.conf:64
> > ```
>
> This has nothing to do with 100m, the error logs have clearly indicated this to you.
>
> the shared dict `prometheus-metrics` has been already defined, see:
>
> https://github.com/apache/apisix/blob/d6c09fba6543c9fc28520ef7172596a01417222f/apisix/cli/ngx_tpl.lua#L171
>
> so you can not define shared dict with the same name(`prometheus-metrics`) in a custom configuration.
I find this key (prometheus-metrics) in config-default.yml, and I want overwrite this value to 100m, so I add these configuration in configmap which will be volumed as config,yml.
https://github.com/apache/apisix/blob/e127cc7a43416673f4b0c77db1ecf09a191abcb8/conf/config-default.yaml#L212-L216
So if I want to overwrite this key (prometheus-metircs) to 100m, where should I modify?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] Applenice commented on issue #3917: request help: question about Prometheus memory consumption
Posted by GitBox <gi...@apache.org>.
Applenice commented on issue #3917:
URL: https://github.com/apache/apisix/issues/3917#issuecomment-874130176
> What about configuring with the original dash style?
Agree, so that it can be consistent with the naming of the plugin.😊
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] Applenice commented on issue #3917: request help: question about Prometheus memory consumption
Posted by GitBox <gi...@apache.org>.
Applenice commented on issue #3917:
URL: https://github.com/apache/apisix/issues/3917#issuecomment-873176056
I was trying to solve this issue and when testing I found the problem:
https://github.com/apache/apisix/blob/e35ef6d54469a312b209bf60b194eca50938ebbe/apisix/cli/ngx_tpl.lua#L144
https://github.com/apache/apisix/blob/e35ef6d54469a312b209bf60b194eca50938ebbe/apisix/cli/ngx_tpl.lua#L145
It can be modified like this:
```
lua_shared_dict internal_status {* http.lua_shared_dict.internal_status *};
```
But it seems to be because this format of `plugin-limit-req` does not conform to the lexical conventions of Lua names. An error is reported:
```
/usr/local/openresty/luajit/bin/luajit: [string "context=... or {}..."]:289: attempt to perform arithmetic on field 'plugin' (a nil value)
```
Can I change the lua_shared_dict variable that uses `-` to `_`? This way the relevant code needs to be modified.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #3917: request help: question about Prometheus memory consumption
Posted by GitBox <gi...@apache.org>.
spacewander commented on issue #3917:
URL: https://github.com/apache/apisix/issues/3917#issuecomment-808043371
https://github.com/apache/apisix/blob/83931ba0a5c95e04ec867a4ffd0821d426ca0d27/apisix/cli/ngx_tpl.lua#L131
I would appreciate it if you can make it configurable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on issue #3917: request help: question about Prometheus memory consumption
Posted by GitBox <gi...@apache.org>.
spacewander commented on issue #3917:
URL: https://github.com/apache/apisix/issues/3917#issuecomment-874090079
What about configuring with the original dash style?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #3917: request help: question about Prometheus memory consumption
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #3917:
URL: https://github.com/apache/apisix/issues/3917#issuecomment-921736339
> And another question is can this completely solve the problem? In my opinion, expanding this value can only alleviate this problem, not solve it. If the memory is full, shouldn't it release part of it?
>
> In my test, this error will continue to appear after the memory is full, and it seems that no request can continue to be recorded. And the memory of the apisix pod will continue to grow, even to 900m. Although I am not sure if it is caused by this problem, it seems that there is something wrong.
>
> 
pls submit another PR and show steps to reproduce
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] suninuni edited a comment on issue #3917: request help: question about Prometheus memory consumption
Posted by GitBox <gi...@apache.org>.
suninuni edited a comment on issue #3917:
URL: https://github.com/apache/apisix/issues/3917#issuecomment-920704670
When I set prometheus-metrics as 100m in configmap apisix, apisix throw an error about key conflicted.
Apisix version is 2.7
```ynl
http:
lua_shared_dicts: # add custom shared cache to nginx.conf
prometheus-metrics: 100m
```
```shell
/usr/local/openresty/luajit/bin/luajit ./apisix/cli/apisix.lua init
WARNING: using fixed Admin API token has security risk.
Please modify "admin_key" in conf/config.yaml .
/usr/local/openresty/luajit/bin/luajit ./apisix/cli/apisix.lua init_etcd
2021/09/16 08:28:56 [emerg] 1#1: the size 104857600 of shared memory zone "prometheus-metrics" conflicts with already declared size 10485760 in /usr/local/apisix/conf/nginx.conf:64
nginx: [emerg] the size 104857600 of shared memory zone "prometheus-metrics" conflicts with already declared size 10485760 in /usr/local/apisix/conf/nginx.conf:64
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on issue #3917: request help: question about Prometheus memory consumption
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #3917:
URL: https://github.com/apache/apisix/issues/3917#issuecomment-873712613
@Applenice Have you tried this:
```yaml
lua_shared_dict internal_status {* http.lua_shared_dict["plugin-limit-req"] *};
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] suninuni edited a comment on issue #3917: request help: question about Prometheus memory consumption
Posted by GitBox <gi...@apache.org>.
suninuni edited a comment on issue #3917:
URL: https://github.com/apache/apisix/issues/3917#issuecomment-921542527
> try
>
> ```
> apisix:
> http:
> lua_shared_dict:
> prometheus-metrics: 10m
> ```
>
> in `config.yaml`
I set this to 100m both in `apisix.http.lua_shared_dict.prometheus-metrics` AND `nginx_config.http.lua_shared_dict.prometheus-metrics` in `config.yml`, but value in conf/nginx.conf is still be 10m.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org