You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by "FeiYing9 (via GitHub)" <gi...@apache.org> on 2025/01/22 09:02:39 UTC
[I] bug: apisix report duplicate metrics [apisix]
FeiYing9 opened a new issue, #11934:
URL: https://github.com/apache/apisix/issues/11934
### Current Behavior
there are several k8s clusters running with apisix, but just one cluster with the problem that apisix report lots of duplicate metrics.
for example:
```bash
apisix_http_status{code="200",route="3fb9d6c2",matched_uri="/api/v1/*",matched_host="xxx",service="",consumer="",node="10.244.19.254",host="xxx",upstream_addr="10.244.19.254:8080",upstream_status="200",uri="/api/v1/cluster_metric/list_task_dimension",method="POST"} 96
apisix_http_status{code="200",route="3fb9d6c2",matched_uri="/api/v1/*",matched_host="xxx",service="",consumer="",node="10.244.19.254",host="xxx",upstream_addr="10.244.19.254:8080",upstream_status="200",uri="/api/v1/cluster_metric/list_task_dimension",method="POST"} 96
...
apisix_http_status{code="200",route="3fb9d6c2",matched_uri="/api/v1/*",matched_host="xxx",service="",consumer="",node="10.244.19.254",host="xxx",upstream_addr="10.244.19.254:8080",upstream_status="200",uri="/api/v1/file/upload",method="POST"} 3188
apisix_http_status{code="200",route="3fb9d6c2",matched_uri="/api/v1/*",matched_host="xxx",service="",consumer="",node="10.244.19.254",host="xxx",upstream_addr="10.244.19.254:8080",upstream_status="200",uri="/api/v1/file/upload",method="POST"} 3188
```
so we will see lots of error logs from prometheus:
```bash
ts=2025-01-22T08:51:08.867Z caller=scrape.go:1793 level=debug component="scrape manager" scrape_pool=serviceMonitor/apisix/apisix/0 target=http://10.244.5.32:9091/apisix/prometheus/metrics msg="Duplicate sample for timestamp" series="apisix_http_latency_bucket{type=\"apisix\",route=\"3fb9d6c2\",service=\"\",consumer=\"\",node=\"10.244.10.60\",host=\"xxx\",upstream_addr=\"10.244.10.60:8080\",upstream_status=\"200\",uri=\"/api/v1/user/routes/ws-f4d69b29-e0a5-44e6-bd92-acf4de9990f0\",method=\"GET\",le=\"100\"}"
```
### Expected Behavior
_No response_
### Error Logs
all error logs is about the shdict:
```bash
2025/01/22 15:07:27 [error] 534#534: *2088505577 [lua] prometheus_resty_counter.lua:39: increasing counter in shdict: lru eviction: key=http_latency_bucket{type="request",route="3fb9d6c2",service="",consumer="",node="10.244.11.36",host="xxx",upstream_addr="10.244.11.36:8080",upstream_status="200",uri="/api/v1/notebook/7eb9852a-be8d-4fac-a593-31f5f7d864b0",method="GET",le="30000.0"}, context: ngx.timer
...
2025/01/22 16:53:00 [error] 499#499: *2098016584 [lua] prometheus.lua:973: log_error(): Shared dictionary used for prometheus metrics is full. REPORTED METRIC DATA MIGHT BE INCOMPLETE. Please increase the size of the dictionary or decrease metric cardinality.; key index: add key: idx=__ngx_prom__key_115158, key=http_latency_bucket{type="request",route="3fb9d6c2",service="",consumer="",node="10.244.11.36",host="xxx",upstream_addr="10.244.11.36:8080",upstream_status="200",uri="/api/v1/project/project-cc83c686-1515-454e-870b-202a20a67727",method="GET",le="Inf"} while logging request, client: 10.245.13.201, server: _, request: "GET /api/v1/project/project-cc83c686-1515-454e-870b-202a20a67727 HTTP/2.0", upstream: "http://10.244.11.36:8080/api/v1/project/project-cc83c686-1515-454e-870b-202a20a67727", host: "qz.sii.edu.cn", referrer: "https://xxx/jobs/distributedTraining?spaceId=ws-f4d69b29-e0a5-44e6-bd92-acf4de9990f0"
```
We accept the issue of insufficient shared dict memory, just hope to know why apisix report duplicate metrics.
### Steps to Reproduce
no ideas
apisix config:
```bash
nginx_config: # config for render the template to genarate nginx.conf
lua_shared_dict:
prometheus-metrics: 200m # yes, it's 200m
...
plugin_attr:
opentelemetry:
set_ngx_var: true
prometheus:
expire: 16
export_addr:
ip: 0.0.0.0
port: 9091
export_uri: /apisix/prometheus/metrics
metric_prefix: apisix_
metrics:
bandwidth:
extra_labels:
- host: $host
- upstream_addr: $upstream_addr
- upstream_status: $upstream_status
- uri: $uri
- method: $request_method
http_latency:
extra_labels:
- host: $host
- upstream_addr: $upstream_addr
- upstream_status: $upstream_status
- uri: $uri
- method: $request_method
http_status:
extra_labels:
- host: $host
- upstream_addr: $upstream_addr
- upstream_status: $upstream_status
- uri: $uri
- method: $request_method
prefer_name: true
```
### Environment
- APISIX version (run `apisix version`): `3.7.0 (helm version: 2.5.0)`
- Operating system (run `uname -a`): `Linux cpu-001 5.4.0-192-generic #212-Ubuntu SMP Fri Jul 5 09:47:39 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux`
- OpenResty / Nginx version (run `openresty -V` or `nginx -V`): `openresty/1.21.4.2`
- k8s version:
```bash
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.16", GitCommit:"cbb86e0d7f4a049666fac0551e8b02ef3d6c3d9a", GitTreeState:"clean", BuildDate:"2024-07-17T01:53:56Z", GoVersion:"go1.22.5", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.16", GitCommit:"cbb86e0d7f4a049666fac0551e8b02ef3d6c3d9a", GitTreeState:"clean", BuildDate:"2024-07-17T01:44:26Z", GoVersion:"go1.22.5", Compiler:"gc", Platform:"linux/amd64"}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: apisix report duplicate metrics [apisix]
Posted by "jorgenll (via GitHub)" <gi...@apache.org>.
jorgenll commented on issue #11934:
URL: https://github.com/apache/apisix/issues/11934#issuecomment-2743145158
Hi,
What is the meaning of expiration in the metrics context? If we don't define an expire value, the consumed RAM memory of the data-plane increase up to the limit but if we define the a value, the cumulative metrics lost data.
It is possible to keep the metric data but expire the old information?
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: apisix report duplicate metrics [apisix]
Posted by "AlinsRan (via GitHub)" <gi...@apache.org>.
AlinsRan closed issue #11934: bug: apisix report duplicate metrics
URL: https://github.com/apache/apisix/issues/11934
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: apisix report duplicate metrics [apisix]
Posted by "yurkovoznyak (via GitHub)" <gi...@apache.org>.
yurkovoznyak commented on issue #11934:
URL: https://github.com/apache/apisix/issues/11934#issuecomment-2607134532
I believe the issue is with metric expiration logic.
I was able to reproduce it locally when I set the expiration time to a low value (like `expire: 10`) and had more than 1 worker process (I tested on 6, but with more workers, it's easier to reproduce).
Removing metrics expiration configuration resolves duplicates
**Environment**
APISIX version (run apisix version): 3.11.0
Operating system (run uname -a): Linux 82435988518b 6.10.14-linuxkit #1 SMP Fri Nov 29 17:22:03 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux
OpenResty / Nginx version (run openresty -V or nginx -V): openresty/1.25.3.2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: apisix report duplicate metrics [apisix]
Posted by "FeiYing9 (via GitHub)" <gi...@apache.org>.
FeiYing9 commented on issue #11934:
URL: https://github.com/apache/apisix/issues/11934#issuecomment-2608696015
> I believe the issue is with metric expiration logic.
>
> I was able to reproduce it locally when I set the expiration time to a low value (like `expire: 10`) and had more than 1 worker process (I tested on 6, but with more workers, it's easier to reproduce).
>
> Removing metrics expiration configuration resolves duplicates
>
> **Environment**
>
> ```
> APISIX version (run apisix version): 3.11.0
> Operating system (run uname -a): Linux 82435988518b 6.10.14-linuxkit #1 SMP Fri Nov 29 17:22:03 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux
> OpenResty / Nginx version (run openresty -V or nginx -V): openresty/1.25.3.2
> ```
thanks for the reply..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] bug: apisix report duplicate metrics [apisix]
Posted by "FeiYing9 (via GitHub)" <gi...@apache.org>.
FeiYing9 commented on issue #11934:
URL: https://github.com/apache/apisix/issues/11934#issuecomment-2608725491
> I believe the issue is with metric expiration logic.
>
> I was able to reproduce it locally when I set the expiration time to a low value (like `expire: 10`) and had more than 1 worker process (I tested on 6, but with more workers, it's easier to reproduce).
>
> Removing metrics expiration configuration resolves duplicates
>
> **Environment**
>
> ```
> APISIX version (run apisix version): 3.11.0
> Operating system (run uname -a): Linux 82435988518b 6.10.14-linuxkit #1 SMP Fri Nov 29 17:22:03 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux
> OpenResty / Nginx version (run openresty -V or nginx -V): openresty/1.25.3.2
> ```
i config the expiration like this:
```yaml
plugin_attr:
prometheus:
expire: 16
metrics:
http_status:
```
then i refer to this [example config](https://github.com/apache/apisix/blob/0cb7d9ac8def4a490c8b49b7755a9484f3018eb2/conf/config.yaml.example#L599) , seem that i make the wrong config of expire, so it's useless?
```yaml
prometheus: # Plugin: prometheus
# metrics: # Create extra labels from nginx variables: https://nginx.org/en/docs/varindex.html
# http_status:
# expire: 0 # The expiration time after which metrics are removed. unit: second.
# # 0 means the metrics will not expire
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org