You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by "Yilialinn (via GitHub)" <gi...@apache.org> on 2024/12/30 08:32:57 UTC
[PR] docs: improve prometheus plugin docs [apisix]
Yilialinn opened a new pull request, #11874:
URL: https://github.com/apache/apisix/pull/11874
### Description
Improve prometheus plugin docs and add Chinese doc.
Fixes # (issue)
### Checklist
- [ ] I have explained the need for this PR and the problem it solves
- [ ] I have explained the changes or the new features added to this PR
- [ ] I have added tests corresponding to this change
- [ ] I have updated the documentation to reflect this change
- [ ] I have verified that this change is backward compatible (If not, please discuss on the [APISIX mailing list](https://github.com/apache/apisix/tree/master#community) first)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs: improve prometheus plugin docs [apisix]
Posted by "guoqqqi (via GitHub)" <gi...@apache.org>.
guoqqqi merged PR #11874:
URL: https://github.com/apache/apisix/pull/11874
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs: improve prometheus plugin docs [apisix]
Posted by "kayx23 (via GitHub)" <gi...@apache.org>.
kayx23 commented on code in PR #11874:
URL: https://github.com/apache/apisix/pull/11874#discussion_r1900607996
##########
docs/zh/latest/plugins/prometheus.md:
##########
@@ -27,408 +27,448 @@ description: 本文将介绍 API 网关 Apache APISIX 如何通过 prometheus
#
-->
+<head>
+ <link rel="canonical" href="https://docs.api7.ai/hub/prometheus" />
+</head>
+
## 描述
-`prometheus` 插件以 [Prometheus 文档](https://prometheus.io/docs/instrumenting/exposition_formats/#exposition-formats)规定的格式上报指标到 Prometheus 中。
+`prometheus` 插件提供将 APISIX 与 Prometheus 集成的能力。
+
+启用该插件后，APISIX 将开始收集相关指标，例如 API 请求和延迟，并以[基于文本的展示格式](https://prometheus.io/docs/instrumenting/exposition_formats/#exposition-formats)导出到 Prometheus。然后，您可以在 Prometheus 中创建事件监控和警报，以监控 API 网关和 API 的健康状况。
+
+## 静态配置
+
+默认情况下，已在默认配置文件 [`config.lua`](https://github.com/apache/apisix/blob/master/apisix/cli/config.lua) 中对 `prometheus` 进行预配置。
+
+要自定义这些值，请将相应的配置添加到 config.yaml 中。例如：
+
+```yaml
+plugin_attr:
+ prometheus: # 插件: prometheus 属性
+ export_uri: /apisix/prometheus/metrics # 设置 Prometheus 指标端点的 URI。
+ metric_prefix: apisix_ # 设置 APISIX 生成的 Prometheus 指标的前缀。
+ enable_export_server: true # 启用 Prometheus 导出服务器。
+ export_addr: # 设置 Prometheus 导出服务器的地址。
+ ip: 127.0.0.1 # 设置 IP。
+ port: 9091 # 设置端口。
+ # metrics: # 为指标创建额外的标签。
+ # http_status: # 这些指标将以 `apisix_` 为前缀。
+ # extra_labels: # 设置 http_status 指标的额外标签。
+ # - upstream_addr: $upstream_addr
+ # - status: $upstream_status
+ # expire: 0 # 指标的过期时间（秒）。
+ # 0 表示指标不会过期。
+ # http_latency:
+ # extra_labels: # 设置 http_latency 指标的额外标签。
+ # - upstream_addr: $upstream_addr
+ # expire: 0 # 指标的过期时间（秒）。
+ # 0 表示指标不会过期。
+ # bandwidth:
+ # extra_labels: # 设置 bandwidth 指标的额外标签。
+ # - upstream_addr: $upstream_addr
+ # expire: 0 # 指标的过期时间（秒）。
+ # 0 表示指标不会过期。
+ # default_buckets: # 设置 `http_latency` 指标直方图的默认桶。
+ # - 10
+ # - 50
+ # - 100
+ # - 200
+ # - 500
+ # - 1000
+ # - 2000
+ # - 5000
+ # - 10000
+ # - 30000
+ # - 60000
+ # - 500
+```
+
+您可以使用 [Nginx 变量](https://nginx.org/en/docs/http/ngx_http_core_module.html)创建 `extra_labels`。请参见[添加额外标签](#add-extra-labels-for-metrics)。
+
+重新加载 APISIX 以使更改生效。
## 属性
| 名称 | 类型 | 必选项 | 默认值 | 描述 |
| ------------ | --------| ------ | ------ | ----------------------------------------------------- |
-| prefer_name | boolean | 否 | false | 当设置为 `true` 时，将使用路由或服务的 `name` 标识请求所命中的路由或服务，否则使用其 `id`。 |
+|`prefer_name` | boolean | 否 | False | 当设置为 `true` 时，则在`prometheus` 指标中导出路由/服务名称而非它们的 `id`。 |
-:::note
+## 指标
-多个路由或服务可以设置为相同的名称，所以当设置 `prefer_name` 为 `true` 时，请规范路由和服务的命名，否则容易引起误解。
+Prometheus 中有不同类型的指标。要了解它们之间的区别，请参见[指标类型](https://prometheus.io/docs/concepts/metric_types/)。
-:::
+以下是 `prometheus` 插件默认导出的指标。有关示例，请参见[获取 APISIX 指标](#get-apisix-metrics)。请注意，一些指标，例如 `apisix_batch_process_entries`，如果没有数据，将不可见。
Review Comment:
ditto, incorrect link
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs: improve prometheus plugin docs [apisix]
Posted by "kayx23 (via GitHub)" <gi...@apache.org>.
kayx23 commented on PR #11874:
URL: https://github.com/apache/apisix/pull/11874#issuecomment-2567383030
Please fix for the above comments and lint errors. Other changes LGTM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs: improve prometheus plugin docs [apisix]
Posted by "kayx23 (via GitHub)" <gi...@apache.org>.
kayx23 commented on code in PR #11874:
URL: https://github.com/apache/apisix/pull/11874#discussion_r1899957344
##########
docs/en/latest/plugins/prometheus.md:
##########
@@ -27,438 +27,423 @@ description: This document contains information about the Apache APISIX promethe
#
-->
-## Description
-
-The `prometheus` Plugin exports metrics in [Prometheus exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/#exposition-formats).
-
-## Attributes
-
-| Name | Type | Required | Default | Description |
-| ----------- | ------- | -------- | ------- | --------------------------------------------------------------------------------- |
-| prefer_name | boolean | False | false | When set to `true`, prints Route/Service name instead of ID in Prometheus metric. |
+<head>
+ <link rel="canonical" href="https://docs.api7.ai/hub/prometheus" />
+</head>
-### Specifying `export_uri`
+## Description
-You can change the default export URI by configuring the `export_uri` attribute under `plugin_attr` in your configuration file (`conf/config.yaml`).
+The `prometheus` plugin provides the capability to integrate APISIX with [Prometheus](https://prometheus.io).
-| Name | Type | Default | Description |
-| ---------- | ------ | ---------------------------- | ------------------------------------- |
-| export_uri | string | "/apisix/prometheus/metrics" | URI to export the Prometheus metrics. |
+After enabling the plugin, APISIX will start collecting relevant metrics, such as API requests and latencies, and exporting them in a [text-based exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/#exposition-formats) to Prometheus. You can then create event monitoring and alerting in Prometheus to monitor the health of your API gateway and APIs.
-Here is a configuration example:
+## Attributes
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- export_uri: /apisix/metrics
+There are different types of metrics in Prometheus. To understand their differences, see [metrics types](https://prometheus.io/docs/concepts/metric_types/).
+
+The following metrics are exported by the `prometheus` plugin by default. See [get APISIX metrics](#get-apisix-metrics) for an example. Note that some metrics, such as `apisix_batch_process_entries`, are not readily visible if there are no data.
+
+
+| Name | Type | Description |
+| ------------------------------ | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| apisix_bandwidth | counter | Total amount of traffic flowing through APISIX in bytes. |
+| apisix_etcd_modify_indexes | gauge | Number of changes to etcd by APISIX keys. |
+| apisix_batch_process_entries | gauge | Number of remaining entries in a batch when sending data in batches, such as with `http logger`, and other logging plugins. |
+| apisix_etcd_reachable | gauge | Whether APISIX can reach etcd. A value of `1` represents reachable and `0` represents unreachable. |
+| apisix_http_status | counter | HTTP status codes returned from upstream services. |
+| apisix_http_requests_total | gauge | Number of HTTP requests from clients. |
+| apisix_nginx_http_current_connections | gauge | Number of current connections with clients. |
+| apisix_nginx_metric_errors_total | counter | Total number of `nginx-lua-prometheus` errors. |
+| apisix_http_latency | histogram | HTTP request latency in milliseconds. |
+| apisix_node_info | gauge | Information of the APISIX node, such as host name. |
+| apisix_shared_dict_capacity_bytes | gauge | The total capacity of an [NGINX shared dictionary](https://github.com/openresty/lua-nginx-module#ngxshareddict). |
+| apisix_shared_dict_free_space_bytes | gauge | The remaining space in an [NGINX shared dictionary](https://github.com/openresty/lua-nginx-module#ngxshareddict). |
+| apisix_upstream_status | gauge | Health check status of upstream nodes, available if health checks are configured on the upstream. A value of `1` represents healthy and `0` represents unhealthy. |
+| apisix_stream_connection_total | counter | Total number of connections handled per stream route. |
+
+## Labels
+
+[Labels](https://prometheus.io/docs/practices/naming/#labels) are attributes of metrics that are used to differentiate metrics.
+
+For example, the `apisix_http_status` metric can be labeled with `route` information to identify which route the HTTP status originates from.
+
+The following are labels for a non-exhaustive list of APISIX metrics and their descriptions.
+
+### Labels for `apisix_http_status`
+
+The following labels are used to differentiate `apisix_http_status` metrics.
+
+| Name | Description |
+| ------------ | ----------------------------------------------------------------------------------------------------------------------------- |
+| code | HTTP response code returned by the upstream node. |
+| route | ID of the route that the HTTP status originates from when `prefer_name` is `false` (default), and name of the route when `prefer_name` to `true`. Default to an empty string if a request does not match any route. |
+| route_id | Available only in Enterprise. ID of the route that the HTTP status originates from regardless of the `prefer_name` setting. |
+| matched_uri | URI of the route that matches the request. Default to an empty string if a request does not match any route. |
+| matched_host | Host of the route that matches the request. Default to an empty string if a request does not match any route, or host is not configured on the route. |
+| service | ID of the service that the HTTP status originates from when `prefer_name` is `false` (default), and name of the service when `prefer_name` to `true`. Default to the configured value of host on the route if the matched route does not belong to any service. |
+| service_id | Available only in Enterprise. ID of the service that the HTTP status originates from regardless of the `prefer_name` setting. |
+| consumer | Name of the consumer associated with a request. Default to an empty string if no consumer is associated with the request. |
+| node | IP address of the upstream node. |
+| gateway_group_id | Available only in Enterprise. ID of the gateway group that the HTTP status originates from. |
+| instance_id | Available only in Enterprise. ID of the gateway instance that the HTTP status originates from. |
+| api_product_id | Available only in Enterprise. Product ID that the HTTP status originates from. |
+
+### Labels for `apisix_bandwidth`
+
+The following labels are used to differentiate `apisix_bandwidth` metrics.
+
+| Name | Description |
+| ---------- | ----------------------------------------------------------------------------------------------------------------------------- |
+| type | Type of traffic, `egress` or `ingress`. |
+| route | ID of the route that bandwidth corresponds to when `prefer_name` is `false` (default), and name of the route when `prefer_name` to `true`. Default to an empty string if a request does not match any route. |
+| route_id | Available only in Enterprise. ID of the route that bandwidth corresponds to regardless of the `prefer_name` setting. |
+| service | ID of the service that bandwidth corresponds to when `prefer_name` is `false` (default), and name of the service when `prefer_name` to `true`. Default to the configured value of host on the route if the matched route does not belong to any service. |
+| service_id | Available only in Enterprise. ID of the service that bandwidth corresponds to regardless of the `prefer_name` setting. |
+| consumer | Name of the consumer associated with a request. Default to an empty string if no consumer is associated with the request. |
+| node | IP address of the upstream node. |
+| gateway_group_id | Available only in Enterprise. ID of the gateway group that bandwidth corresponds to. |
+| instance_id | Available only in Enterprise. ID of the gateway instance that bandwidth corresponds to. |
+| api_product_id | Available only in Enterprise. Product ID that bandwidth corresponds to. |
+
+### Labels for `apisix_http_latency`
+
+The following labels are used to differentiate `apisix_http_latency` metrics.
+
+| Name | Description |
+| ---------- | ----------------------------------------------------------------------------------------------------------------------------------- |
+| type | Type of latencies. See [latency types](#latency-types) for details. |
+| route | ID of the route that latencies correspond to when `prefer_name` is `false` (default), and name of the route when `prefer_name` to `true`. Default to an empty string if a request does not match any route. |
+| route_id | Available only in Enterprise. ID of the route that latencies correspond to regardless of the `prefer_name` setting. |
+| service | ID of the service that latencies correspond to when `prefer_name` is `false` (default), and name of the service when `prefer_name` to `true`. Default to the configured value of host on the route if the matched route does not belong to any service. |
+| service_id | Available only in Enterprise. ID of the service that latencies correspond to regardless of the `prefer_name` setting. |
+| consumer | Name of the consumer associated with latencies. Default to an empty string if no consumer is associated with the request. |
+| node | IP address of the upstream node associated with latencies. |
+| gateway_group_id | Available only in Enterprise. ID of the gateway group that latencies correspond to. |
+| instance_id | Available only in Enterprise. ID of the gateway instance that latencies correspond to. |
+| api_product_id | Available only in Enterprise. Product ID that latencies correspond to. |
+
+#### Latency Types
+
+`apisix_http_latency` can be labeled with one of the three types:
+
+* `request` represents the time elapsed between the first byte was read from the client and the log write after the last byte was sent to the client.
+
+* `upstream` represents the time elapsed waiting on responses from the upstream service.
+
+* `apisix` represents the difference between the `request` latency and `upstream` latency.
+
+In other words, the APISIX latency is not only attributed to the Lua processing. It should be understood as follows:
+
+```text
+APISIX latency
+ = downstream request time - upstream response time
+ = downstream traffic latency + NGINX latency
```
-### Specifying `metrics`
-
-For http request related metrics, you could specify extra labels, which match the APISIX variables.
+### Labels for `apisix_upstream_status`
-If you specify label for nonexist APISIX variable, the label value would be "".
+The following labels are used to differentiate `apisix_upstream_status` metrics.
-Currently, only below metrics are supported:
+| Name | Description |
+| ---------- | --------------------------------------------------------------------------------------------------- |
+| name | Resource ID corresponding to the upstream configured with health checks, such as `/apisix/routes/1` and `/apisix/upstreams/1`. |
+| ip | IP address of the upstream node. |
+| port | Port number of the node. |
-* http_status
-* http_latency
-* bandwidth
+## Examples
-Here is a configuration example:
-
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- metrics:
- http_status:
- extra_labels:
- - upstream_addr: $upstream_addr
- - upstream_status: $upstream_status
-```
+The examples below demonstrate how you can work with the `prometheus` plugin for different scenarios.
-### Specifying `default_buckets`
+### Get APISIX Metrics
-`DEFAULT_BUCKETS` is the default value for bucket array in `http_latency` metrics.
+The following example demonstrates how you can get metrics from APISIX.
-You can change the `DEFAULT_BUCKETS` by configuring `default_buckets` attribute in you configuration file.
+The default Prometheus metrics endpoint and other Prometheus related configurations can be found in the [static configuration](#static-configurations). If you would like to customize these configuration, see [configuration files](/apisix/reference/configuration-files#configyaml-and-configyamlexample).
Review Comment:
It's in `config.yaml`. You can mention it in a separate section - this is what other plugin docs here do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs: improve prometheus plugin docs [apisix]
Posted by "kayx23 (via GitHub)" <gi...@apache.org>.
kayx23 commented on code in PR #11874:
URL: https://github.com/apache/apisix/pull/11874#discussion_r1900451840
##########
docs/en/latest/plugins/prometheus.md:
##########
@@ -27,439 +27,471 @@ description: This document contains information about the Apache APISIX promethe
#
-->
-## Description
-
-The `prometheus` Plugin exports metrics in [Prometheus exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/#exposition-formats).
+<head>
+ <link rel="canonical" href="https://docs.api7.ai/hub/prometheus" />
+</head>
-## Attributes
+## Description
-| Name | Type | Required | Default | Description |
-| ----------- | ------- | -------- | ------- | --------------------------------------------------------------------------------- |
-| prefer_name | boolean | False | false | When set to `true`, prints Route/Service name instead of ID in Prometheus metric. |
+The `prometheus` Plugin provides the capability to integrate APISIX with [Prometheus](https://prometheus.io).
-### Specifying `export_uri`
+After enabling the Plugin, APISIX will start collecting relevant metrics, such as API requests and latencies, and exporting them in a [text-based exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/#exposition-formats) to Prometheus. You can then create event monitoring and alerting in Prometheus to monitor the health of your API gateway and APIs.
-You can change the default export URI by configuring the `export_uri` attribute under `plugin_attr` in your configuration file (`conf/config.yaml`).
+## Static Configurations
-| Name | Type | Default | Description |
-| ---------- | ------ | ---------------------------- | ------------------------------------- |
-| export_uri | string | "/apisix/prometheus/metrics" | URI to export the Prometheus metrics. |
+By default, `prometheus` configurations are pre-configured in the [default configuration](https://github.com/apache/apisix/blob/master/apisix/cli/config.lua).
-Here is a configuration example:
+To customize these values, add the corresponding configurations to `config.yaml`. For example:
-```yaml title="conf/config.yaml"
+```yaml
plugin_attr:
- prometheus:
- export_uri: /apisix/metrics
+ prometheus: # Plugin: prometheus attributes
+ export_uri: /apisix/prometheus/metrics # Set the URI for the Prometheus metrics endpoint.
+ metric_prefix: apisix_ # Set the prefix for Prometheus metrics generated by APISIX.
+ enable_export_server: true # Enable the Prometheus export server.
+ export_addr: # Set the address for the Prometheus export server.
+ ip: 127.0.0.1 # Set the IP.
+ port: 9091 # Set the port.
+ # metrics: # Create extra labels for metrics.
+ # http_status: # These metrics will be prefixed with `apisix_`.
+ # extra_labels: # Set the extra labels for http_status metrics.
+ # - upstream_addr: $upstream_addr
+ # - status: $upstream_status
+ # expire: 0 # The expiration time of metrics in seconds.
+ # 0 means the metrics will not expire.
+ # http_latency:
+ # extra_labels: # Set the extra labels for http_latency metrics.
+ # - upstream_addr: $upstream_addr
+ # expire: 0 # The expiration time of metrics in seconds.
+ # 0 means the metrics will not expire.
+ # bandwidth:
+ # extra_labels: # Set the extra labels for bandwidth metrics.
+ # - upstream_addr: $upstream_addr
+ # expire: 0 # The expiration time of metrics in seconds.
+ # 0 means the metrics will not expire.
+ # default_buckets: # Set the default buckets for the `http_latency` metrics histogram.
+ # - 10
+ # - 50
+ # - 100
+ # - 200
+ # - 500
+ # - 1000
+ # - 2000
+ # - 5000
+ # - 10000
+ # - 30000
+ # - 60000
+ # - 500
```
-### Specifying `metrics`
+You can use the [Nginx variable](https://nginx.org/en/docs/http/ngx_http_core_module.html) to create `extra_labels`. See [add extra labels](/hub/prometheus#add-extra-labels-for-metrics).
Review Comment:
```suggestion
You can use the [Nginx variable](https://nginx.org/en/docs/http/ngx_http_core_module.html) to create `extra_labels`. See [add extra labels](#add-extra-labels-for-metrics).
```
##########
docs/en/latest/plugins/prometheus.md:
##########
@@ -27,439 +27,471 @@ description: This document contains information about the Apache APISIX promethe
#
-->
-## Description
-
-The `prometheus` Plugin exports metrics in [Prometheus exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/#exposition-formats).
+<head>
+ <link rel="canonical" href="https://docs.api7.ai/hub/prometheus" />
+</head>
-## Attributes
+## Description
-| Name | Type | Required | Default | Description |
-| ----------- | ------- | -------- | ------- | --------------------------------------------------------------------------------- |
-| prefer_name | boolean | False | false | When set to `true`, prints Route/Service name instead of ID in Prometheus metric. |
+The `prometheus` Plugin provides the capability to integrate APISIX with [Prometheus](https://prometheus.io).
-### Specifying `export_uri`
+After enabling the Plugin, APISIX will start collecting relevant metrics, such as API requests and latencies, and exporting them in a [text-based exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/#exposition-formats) to Prometheus. You can then create event monitoring and alerting in Prometheus to monitor the health of your API gateway and APIs.
-You can change the default export URI by configuring the `export_uri` attribute under `plugin_attr` in your configuration file (`conf/config.yaml`).
+## Static Configurations
-| Name | Type | Default | Description |
-| ---------- | ------ | ---------------------------- | ------------------------------------- |
-| export_uri | string | "/apisix/prometheus/metrics" | URI to export the Prometheus metrics. |
+By default, `prometheus` configurations are pre-configured in the [default configuration](https://github.com/apache/apisix/blob/master/apisix/cli/config.lua).
-Here is a configuration example:
+To customize these values, add the corresponding configurations to `config.yaml`. For example:
-```yaml title="conf/config.yaml"
+```yaml
plugin_attr:
- prometheus:
- export_uri: /apisix/metrics
+ prometheus: # Plugin: prometheus attributes
+ export_uri: /apisix/prometheus/metrics # Set the URI for the Prometheus metrics endpoint.
+ metric_prefix: apisix_ # Set the prefix for Prometheus metrics generated by APISIX.
+ enable_export_server: true # Enable the Prometheus export server.
+ export_addr: # Set the address for the Prometheus export server.
+ ip: 127.0.0.1 # Set the IP.
+ port: 9091 # Set the port.
+ # metrics: # Create extra labels for metrics.
+ # http_status: # These metrics will be prefixed with `apisix_`.
+ # extra_labels: # Set the extra labels for http_status metrics.
+ # - upstream_addr: $upstream_addr
+ # - status: $upstream_status
+ # expire: 0 # The expiration time of metrics in seconds.
+ # 0 means the metrics will not expire.
+ # http_latency:
+ # extra_labels: # Set the extra labels for http_latency metrics.
+ # - upstream_addr: $upstream_addr
+ # expire: 0 # The expiration time of metrics in seconds.
+ # 0 means the metrics will not expire.
+ # bandwidth:
+ # extra_labels: # Set the extra labels for bandwidth metrics.
+ # - upstream_addr: $upstream_addr
+ # expire: 0 # The expiration time of metrics in seconds.
+ # 0 means the metrics will not expire.
+ # default_buckets: # Set the default buckets for the `http_latency` metrics histogram.
+ # - 10
+ # - 50
+ # - 100
+ # - 200
+ # - 500
+ # - 1000
+ # - 2000
+ # - 5000
+ # - 10000
+ # - 30000
+ # - 60000
+ # - 500
```
-### Specifying `metrics`
+You can use the [Nginx variable](https://nginx.org/en/docs/http/ngx_http_core_module.html) to create `extra_labels`. See [add extra labels](/hub/prometheus#add-extra-labels-for-metrics).
-For http request related metrics, you could specify extra labels, which match the APISIX variables.
+Reload APISIX for changes to take effect.
-If you specify label for nonexist APISIX variable, the label value would be "".
+## Attribute
-Currently, only below metrics are supported:
+| Name | Type | Required | Default | Valid values | Description |
+| ------------- | ------- | -------- | ------- | ------------ | ------------------------------------------ |
+| `prefer_name` | boolean | | False | | If true, export Route/Service name instead of their ID in Prometheus metrics. |
-* http_status
-* http_latency
-* bandwidth
+## Metrics
-Here is a configuration example:
+There are different types of metrics in Prometheus. To understand their differences, see [metrics types](https://prometheus.io/docs/concepts/metric_types/).
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- metrics:
- http_status:
- extra_labels:
- - upstream_addr: $upstream_addr
- - upstream_status: $upstream_status
-```
+The following metrics are exported by the `prometheus` Plugin by default. See [get APISIX metrics](#get-apisix-metrics) for an example. Note that some metrics, such as `apisix_batch_process_entries`, are not readily visible if there are no data.
-### Specifying `default_buckets`
+| Name | Type | Description |
+| ------------------------------ | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| apisix_bandwidth | counter | Total amount of traffic flowing through APISIX in bytes. |
+| apisix_etcd_modify_indexes | gauge | Number of changes to etcd by APISIX keys. |
+| apisix_batch_process_entries | gauge | Number of remaining entries in a batch when sending data in batches, such as with `http logger`, and other logging Plugins. |
+| apisix_etcd_reachable | gauge | Whether APISIX can reach etcd. A value of `1` represents reachable and `0` represents unreachable. |
+| apisix_http_status | counter | HTTP status codes returned from upstream Services. |
+| apisix_http_requests_total | gauge | Number of HTTP requests from clients. |
+| apisix_nginx_http_current_connections | gauge | Number of current connections with clients. |
+| apisix_nginx_metric_errors_total | counter | Total number of `nginx-lua-prometheus` errors. |
+| apisix_http_latency | histogram | HTTP request latency in milliseconds. |
+| apisix_node_info | gauge | Information of the APISIX node, such as host name. |
+| apisix_shared_dict_capacity_bytes | gauge | The total capacity of an [NGINX shared dictionary](https://github.com/openresty/lua-nginx-module#ngxshareddict). |
+| apisix_shared_dict_free_space_bytes | gauge | The remaining space in an [NGINX shared dictionary](https://github.com/openresty/lua-nginx-module#ngxshareddict). |
+| apisix_upstream_status | gauge | Health check status of upstream nodes, available if health checks are configured on the upstream. A value of `1` represents healthy and `0` represents unhealthy. |
+| apisix_stream_connection_total | counter | Total number of connections handled per stream Route. |
-`DEFAULT_BUCKETS` is the default value for bucket array in `http_latency` metrics.
+## Labels
-You can change the `DEFAULT_BUCKETS` by configuring `default_buckets` attribute in you configuration file.
+[Labels](https://prometheus.io/docs/practices/naming/#labels) are attributes of metrics that are used to differentiate metrics.
-Here is a configuration example:
+For example, the `apisix_http_status` metric can be labeled with `route` information to identify which Route the HTTP status originates from.
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- default_buckets:
- - 15
- - 55
- - 105
- - 205
- - 505
-```
+The following are labels for a non-exhaustive list of APISIX metrics and their descriptions.
-### Specifying `expire`
+### Labels for `apisix_http_status`
-`expire` sets the expiration time of `apisix_http_status`, `apisix_bandwidth`, and `apisix_http_latency` metrics in seconds. When set to 0, metrics will not expire.
+The following labels are used to differentiate `apisix_http_status` metrics.
-Here is a configuration example:
+| Name | Description |
+| ------------ | ----------------------------------------------------------------------------------------------------------------------------- |
+| code | HTTP response code returned by the upstream node. |
+| route | ID of the Route that the HTTP status originates from when `prefer_name` is `false` (default), and name of the Route when `prefer_name` to `true`. Default to an empty string if a request does not match any Route. |
+| route_id | Available only in Enterprise. ID of the Route that the HTTP status originates from regardless of the `prefer_name` setting. |
+| matched_uri | URI of the Route that matches the request. Default to an empty string if a request does not match any Route. |
+| matched_host | Host of the Route that matches the request. Default to an empty string if a request does not match any Route, or host is not configured on the Route. |
+| service | ID of the Service that the HTTP status originates from when `prefer_name` is `false` (default), and name of the Service when `prefer_name` to `true`. Default to the configured value of host on the Route if the matched Route does not belong to any Service. |
+| service_id | Available only in Enterprise. ID of the Service that the HTTP status originates from regardless of the `prefer_name` setting. |
+| consumer | Name of the Consumer associated with a request. Default to an empty string if no Consumer is associated with the request. |
+| node | IP address of the upstream node. |
+| gateway_group_id | Available only in Enterprise. ID of the gateway group that the HTTP status originates from. |
+| instance_id | Available only in Enterprise. ID of the gateway instance that the HTTP status originates from. |
+| api_product_id | Available only in Enterprise. Product ID that the HTTP status originates from. |
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- expire: 86400
-```
+### Labels for `apisix_bandwidth`
-## Metrics endpoint
+The following labels are used to differentiate `apisix_bandwidth` metrics.
-This Plugin will add the metrics endpoint `/apisix/prometheus/metrics` or your custom export URI for exposing the metrics.
+| Name | Description |
+| ---------- | ----------------------------------------------------------------------------------------------------------------------------- |
+| type | Type of traffic, `egress` or `ingress`. |
+| route | ID of the Route that bandwidth corresponds to when `prefer_name` is `false` (default), and name of the Route when `prefer_name` to `true`. Default to an empty string if a request does not match any Route. |
+| route_id | Available only in Enterprise. ID of the Route that bandwidth corresponds to regardless of the `prefer_name` setting. |
+| service | ID of the Service that bandwidth corresponds to when `prefer_name` is `false` (default), and name of the Service when `prefer_name` to `true`. Default to the configured value of host on the Route if the matched Route does not belong to any Service. |
+| service_id | Available only in Enterprise. ID of the Service that bandwidth corresponds to regardless of the `prefer_name` setting. |
+| consumer | Name of the Consumer associated with a request. Default to an empty string if no Consumer is associated with the request. |
+| node | IP address of the upstream node. |
+| gateway_group_id | Available only in Enterprise. ID of the gateway group that bandwidth corresponds to. |
+| instance_id | Available only in Enterprise. ID of the gateway instance that bandwidth corresponds to. |
+| api_product_id | Available only in Enterprise. Product ID that bandwidth corresponds to. |
-These metrics are exposed by a separate Prometheus server address. By default, the address is `127.0.0.1:9091`. You can change it in your configuration file (`conf/config.yaml`):
+### Labels for `apisix_http_latency`
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- export_addr:
- ip: ${{INTRANET_IP}}
- port: 9092
-```
+The following labels are used to differentiate `apisix_http_latency` metrics.
-Now, if the environment variable `INTRANET_IP` is `172.1.1.1`, APISIX will export the metrics via `172.1.1.1:9092`.
+| Name | Description |
+| ---------- | ----------------------------------------------------------------------------------------------------------------------------------- |
+| type | Type of latencies. See [latency types](#latency-types) for details. |
+| route | ID of the Route that latencies correspond to when `prefer_name` is `false` (default), and name of the Route when `prefer_name` to `true`. Default to an empty string if a request does not match any Route. |
+| route_id | Available only in Enterprise. ID of the Route that latencies correspond to regardless of the `prefer_name` setting. |
+| service | ID of the Service that latencies correspond to when `prefer_name` is `false` (default), and name of the Service when `prefer_name` to `true`. Default to the configured value of host on the Route if the matched Route does not belong to any Service. |
+| service_id | Available only in Enterprise. ID of the Service that latencies correspond to regardless of the `prefer_name` setting. |
+| consumer | Name of the Consumer associated with latencies. Default to an empty string if no Consumer is associated with the request. |
+| node | IP address of the upstream node associated with latencies. |
+| gateway_group_id | Available only in Enterprise. ID of the gateway group that latencies correspond to. |
+| instance_id | Available only in Enterprise. ID of the gateway instance that latencies correspond to. |
+| api_product_id | Available only in Enterprise. Product ID that latencies correspond to. |
-If you still want to expose the metrics via the data plane port (default: `9080`), you can configure it as shown below:
+#### Latency Types
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- enable_export_server: false
+`apisix_http_latency` can be labeled with one of the three types:
+
+* `request` represents the time elapsed between the first byte was read from the client and the log write after the last byte was sent to the client.
+
+* `upstream` represents the time elapsed waiting on responses from the upstream Service.
+
+* `apisix` represents the difference between the `request` latency and `upstream` latency.
+
+In other words, the APISIX latency is not only attributed to the Lua processing. It should be understood as follows:
+
+```text
+APISIX latency
+ = downstream request time - upstream response time
+ = downstream traffic latency + NGINX latency
```
-You can then expose it by using the [public-api](public-api.md) Plugin.
+### Labels for `apisix_upstream_status`
-:::info IMPORTANT
+The following labels are used to differentiate `apisix_upstream_status` metrics.
-If the Prometheus plugin collects too many metrics, it will take CPU resources to calculate the metric data when getting the metrics via URI, which may affect APISIX to process normal requests. To solve this problem, APISIX exposes the URI and calculates the metrics in the [privileged agent](https://github.com/openresty/lua-resty-core/blob/master/lib/ngx/process.md#enable_privileged_agent).
-If the URI is exposed using the public-api plugin, then APISIX will calculate the metric data in a normal worker process, which may still affect APISIX processing of normal requests.
+| Name | Description |
+| ---------- | --------------------------------------------------------------------------------------------------- |
+| name | Resource ID corresponding to the upstream configured with health checks, such as `/apisix/routes/1` and `/apisix/upstreams/1`. |
+| ip | IP address of the upstream node. |
+| port | Port number of the node. |
-This feature requires APISIX to run on [APISIX-Runtime](../FAQ.md#how-do-i-build-the-apisix-runtime-environment).
+## Examples
-:::
+The examples below demonstrate how you can work with the `prometheus` Plugin for different scenarios.
-## Enable Plugin
+### Get APISIX Metrics
-The `prometheus` Plugin can be enabled with an empty table.
+The following example demonstrates how you can get metrics from APISIX.
-The example below shows how you can configure the Plugin on a specific Route:
+The default Prometheus metrics endpoint and other Prometheus related configurations can be found in the [static configuration](#static-configurations). If you would like to customize these configuration, see [configuration based on environments](https://apisix.apache.org/docs/apisix/profile/).
-:::note
-You can fetch the `admin_key` from `config.yaml` and save to an environment variable with the following command:
+If you deploy APISIX in a containerized environment and would like to access the Prometheus metrics endpoint externally, update the configuration file as follows and reload APISIX:
-```bash
-admin_key=$(yq '.deployment.admin.admin_key[0].key' conf/config.yaml | sed 's/"//g')
+```yaml title="conf/config.yaml"
+plugin_attr:
+ prometheus:
+ export_addr:
+ ip: 0.0.0.0
```
-:::
+Send a request to the APISIX Prometheus metrics endpoint:
```shell
-curl http://127.0.0.1:9180/apisix/admin/routes/1 -H "X-API-KEY: $admin_key" -X PUT -d '
-{
- "uri": "/hello",
- "plugins": {
- "prometheus":{}
- },
- "upstream": {
- "type": "roundrobin",
- "nodes": {
- "127.0.0.1:80": 1
- }
- }
-}'
+curl "http://127.0.0.1:9091/apisix/prometheus/metrics"
```
-:::note
-
-When `prefer_name` is set to `true` make sure to not duplicate names for multiple Routes/Services or it could be misleading.
-
-:::
+You should see an output similar to the following:
-<!-- You can use [APISIX Dashboard](https://github.com/apache/apisix-dashboard) to complete the above operations through the web console.
-
-First, add a Route:
+```text
+# HELP apisix_bandwidth Total bandwidth in bytes consumed per Service in Apisix
+# TYPE apisix_bandwidth counter
+apisix_bandwidth{type="egress",route="",service="",consumer="",node=""} 8417
+apisix_bandwidth{type="egress",route="1",service="",consumer="",node="127.0.0.1"} 1420
+apisix_bandwidth{type="egress",route="2",service="",consumer="",node="127.0.0.1"} 1420
+apisix_bandwidth{type="ingress",route="",service="",consumer="",node=""} 189
+apisix_bandwidth{type="ingress",route="1",service="",consumer="",node="127.0.0.1"} 332
+apisix_bandwidth{type="ingress",route="2",service="",consumer="",node="127.0.0.1"} 332
+# HELP apisix_etcd_modify_indexes Etcd modify index for APISIX keys
+# TYPE apisix_etcd_modify_indexes gauge
+apisix_etcd_modify_indexes{key="consumers"} 0
+apisix_etcd_modify_indexes{key="global_rules"} 0
+...
+```
-
+### Expose APISIX Metrics on Public API Endpoint
-Then add prometheus plugin:
+The following example demonstrates how you can disable the Prometheus export server that, by default, exposes an endpoint on port `9091`, and expose APISIX Prometheus metrics on a new public API endpoint on port `9080`, which APISIX uses to listen to other client requests.
- -->
+Disable the Prometheus export server in the configuration file and reload APISIX for changes to take effect:
-## Fetching metrics
+```yaml title="conf/config.yaml"
+plugin_attr:
+ prometheus:
+ enable_export_server: false
+```
-You can fetch the metrics from the specified export URI (default: `/apisix/prometheus/metrics`):
+Next, create a Route with [`public-api`](./public-api.md) Plugin and expose a public API endpoint for APISIX metrics:
```shell
-curl -i http://127.0.0.1:9091/apisix/prometheus/metrics
+curl "http://127.0.0.1:9180/apisix/admin/routes/prometheus-metrics" -X PUT \
+ -H "X-API-KEY: ${admin_key}" \
+ -d '{
+ "uri": "/apisix/prometheus/metrics",
+ "plugins": {
+ "public-api": {}
+ }
+ }'
```
-You can add this address to Prometheus to fetch the data:
+Send a request to the new metrics endpoint to verify:
-```yaml
-scrape_configs:
- - job_name: "apisix"
- scrape_interval: 15s # This value will be related to the time range of the rate function in Prometheus QL. The time range in the rate function should be at least twice this value.
- metrics_path: "/apisix/prometheus/metrics"
- static_configs:
- - targets: ["127.0.0.1:9091"]
+```shell
+curl "http://127.0.0.1:9080/apisix/prometheus/metrics"
```
-Now, you will be able to check the status in your Prometheus console:
-
-
+You should see an output similar to the following:
-
-
-## Using Grafana to graph the metrics
-
-Metrics exported by the `prometheus` Plugin can be graphed in Grafana using a drop in dashboard.
-
-To set it up, download [Grafana dashboard meta](https://github.com/apache/apisix/blob/master/docs/assets/other/json/apisix-grafana-dashboard.json) and import it in Grafana. Or, you can go to [Grafana official](https://grafana.com/grafana/dashboards/11719) for Grafana metadata.
+```text
+# HELP apisix_http_requests_total The total number of client requests since APISIX started
+# TYPE apisix_http_requests_total gauge
+apisix_http_requests_total 1
+# HELP apisix_nginx_http_current_connections Number of HTTP connections
+# TYPE apisix_nginx_http_current_connections gauge
+apisix_nginx_http_current_connections{state="accepted"} 1
+apisix_nginx_http_current_connections{state="active"} 1
+apisix_nginx_http_current_connections{state="handled"} 1
+apisix_nginx_http_current_connections{state="reading"} 0
+apisix_nginx_http_current_connections{state="waiting"} 0
+apisix_nginx_http_current_connections{state="writing"} 1
+...
+```
-
+### Integrate APISIX with Prometheus and Grafana
-
+To learn about how to collect APISIX metrics with Prometheus and visualize them in Grafana, see [how-to guide](/apisix/how-to-guide/observability/monitor-apisix-with-prometheus).
Review Comment:
Broken link as the doc is not available here. Maybe just remove this example.
##########
docs/en/latest/plugins/prometheus.md:
##########
@@ -27,439 +27,471 @@ description: This document contains information about the Apache APISIX promethe
#
-->
-## Description
-
-The `prometheus` Plugin exports metrics in [Prometheus exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/#exposition-formats).
+<head>
+ <link rel="canonical" href="https://docs.api7.ai/hub/prometheus" />
+</head>
-## Attributes
+## Description
-| Name | Type | Required | Default | Description |
-| ----------- | ------- | -------- | ------- | --------------------------------------------------------------------------------- |
-| prefer_name | boolean | False | false | When set to `true`, prints Route/Service name instead of ID in Prometheus metric. |
+The `prometheus` Plugin provides the capability to integrate APISIX with [Prometheus](https://prometheus.io).
-### Specifying `export_uri`
+After enabling the Plugin, APISIX will start collecting relevant metrics, such as API requests and latencies, and exporting them in a [text-based exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/#exposition-formats) to Prometheus. You can then create event monitoring and alerting in Prometheus to monitor the health of your API gateway and APIs.
-You can change the default export URI by configuring the `export_uri` attribute under `plugin_attr` in your configuration file (`conf/config.yaml`).
+## Static Configurations
-| Name | Type | Default | Description |
-| ---------- | ------ | ---------------------------- | ------------------------------------- |
-| export_uri | string | "/apisix/prometheus/metrics" | URI to export the Prometheus metrics. |
+By default, `prometheus` configurations are pre-configured in the [default configuration](https://github.com/apache/apisix/blob/master/apisix/cli/config.lua).
-Here is a configuration example:
+To customize these values, add the corresponding configurations to `config.yaml`. For example:
-```yaml title="conf/config.yaml"
+```yaml
plugin_attr:
- prometheus:
- export_uri: /apisix/metrics
+ prometheus: # Plugin: prometheus attributes
+ export_uri: /apisix/prometheus/metrics # Set the URI for the Prometheus metrics endpoint.
+ metric_prefix: apisix_ # Set the prefix for Prometheus metrics generated by APISIX.
+ enable_export_server: true # Enable the Prometheus export server.
+ export_addr: # Set the address for the Prometheus export server.
+ ip: 127.0.0.1 # Set the IP.
+ port: 9091 # Set the port.
+ # metrics: # Create extra labels for metrics.
+ # http_status: # These metrics will be prefixed with `apisix_`.
+ # extra_labels: # Set the extra labels for http_status metrics.
+ # - upstream_addr: $upstream_addr
+ # - status: $upstream_status
+ # expire: 0 # The expiration time of metrics in seconds.
+ # 0 means the metrics will not expire.
+ # http_latency:
+ # extra_labels: # Set the extra labels for http_latency metrics.
+ # - upstream_addr: $upstream_addr
+ # expire: 0 # The expiration time of metrics in seconds.
+ # 0 means the metrics will not expire.
+ # bandwidth:
+ # extra_labels: # Set the extra labels for bandwidth metrics.
+ # - upstream_addr: $upstream_addr
+ # expire: 0 # The expiration time of metrics in seconds.
+ # 0 means the metrics will not expire.
+ # default_buckets: # Set the default buckets for the `http_latency` metrics histogram.
+ # - 10
+ # - 50
+ # - 100
+ # - 200
+ # - 500
+ # - 1000
+ # - 2000
+ # - 5000
+ # - 10000
+ # - 30000
+ # - 60000
+ # - 500
```
-### Specifying `metrics`
+You can use the [Nginx variable](https://nginx.org/en/docs/http/ngx_http_core_module.html) to create `extra_labels`. See [add extra labels](/hub/prometheus#add-extra-labels-for-metrics).
-For http request related metrics, you could specify extra labels, which match the APISIX variables.
+Reload APISIX for changes to take effect.
-If you specify label for nonexist APISIX variable, the label value would be "".
+## Attribute
-Currently, only below metrics are supported:
+| Name | Type | Required | Default | Valid values | Description |
+| ------------- | ------- | -------- | ------- | ------------ | ------------------------------------------ |
+| `prefer_name` | boolean | | False | | If true, export Route/Service name instead of their ID in Prometheus metrics. |
-* http_status
-* http_latency
-* bandwidth
+## Metrics
-Here is a configuration example:
+There are different types of metrics in Prometheus. To understand their differences, see [metrics types](https://prometheus.io/docs/concepts/metric_types/).
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- metrics:
- http_status:
- extra_labels:
- - upstream_addr: $upstream_addr
- - upstream_status: $upstream_status
-```
+The following metrics are exported by the `prometheus` Plugin by default. See [get APISIX metrics](#get-apisix-metrics) for an example. Note that some metrics, such as `apisix_batch_process_entries`, are not readily visible if there are no data.
-### Specifying `default_buckets`
+| Name | Type | Description |
+| ------------------------------ | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| apisix_bandwidth | counter | Total amount of traffic flowing through APISIX in bytes. |
+| apisix_etcd_modify_indexes | gauge | Number of changes to etcd by APISIX keys. |
+| apisix_batch_process_entries | gauge | Number of remaining entries in a batch when sending data in batches, such as with `http logger`, and other logging Plugins. |
+| apisix_etcd_reachable | gauge | Whether APISIX can reach etcd. A value of `1` represents reachable and `0` represents unreachable. |
+| apisix_http_status | counter | HTTP status codes returned from upstream Services. |
+| apisix_http_requests_total | gauge | Number of HTTP requests from clients. |
+| apisix_nginx_http_current_connections | gauge | Number of current connections with clients. |
+| apisix_nginx_metric_errors_total | counter | Total number of `nginx-lua-prometheus` errors. |
+| apisix_http_latency | histogram | HTTP request latency in milliseconds. |
+| apisix_node_info | gauge | Information of the APISIX node, such as host name. |
+| apisix_shared_dict_capacity_bytes | gauge | The total capacity of an [NGINX shared dictionary](https://github.com/openresty/lua-nginx-module#ngxshareddict). |
+| apisix_shared_dict_free_space_bytes | gauge | The remaining space in an [NGINX shared dictionary](https://github.com/openresty/lua-nginx-module#ngxshareddict). |
+| apisix_upstream_status | gauge | Health check status of upstream nodes, available if health checks are configured on the upstream. A value of `1` represents healthy and `0` represents unhealthy. |
+| apisix_stream_connection_total | counter | Total number of connections handled per stream Route. |
Review Comment:
Replace all `stream Route` with `Stream Route`. There are about 7 instances in this doc.
##########
docs/en/latest/plugins/prometheus.md:
##########
@@ -27,439 +27,471 @@ description: This document contains information about the Apache APISIX promethe
#
-->
-## Description
-
-The `prometheus` Plugin exports metrics in [Prometheus exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/#exposition-formats).
+<head>
+ <link rel="canonical" href="https://docs.api7.ai/hub/prometheus" />
+</head>
-## Attributes
+## Description
-| Name | Type | Required | Default | Description |
-| ----------- | ------- | -------- | ------- | --------------------------------------------------------------------------------- |
-| prefer_name | boolean | False | false | When set to `true`, prints Route/Service name instead of ID in Prometheus metric. |
+The `prometheus` Plugin provides the capability to integrate APISIX with [Prometheus](https://prometheus.io).
-### Specifying `export_uri`
+After enabling the Plugin, APISIX will start collecting relevant metrics, such as API requests and latencies, and exporting them in a [text-based exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/#exposition-formats) to Prometheus. You can then create event monitoring and alerting in Prometheus to monitor the health of your API gateway and APIs.
-You can change the default export URI by configuring the `export_uri` attribute under `plugin_attr` in your configuration file (`conf/config.yaml`).
+## Static Configurations
-| Name | Type | Default | Description |
-| ---------- | ------ | ---------------------------- | ------------------------------------- |
-| export_uri | string | "/apisix/prometheus/metrics" | URI to export the Prometheus metrics. |
+By default, `prometheus` configurations are pre-configured in the [default configuration](https://github.com/apache/apisix/blob/master/apisix/cli/config.lua).
-Here is a configuration example:
+To customize these values, add the corresponding configurations to `config.yaml`. For example:
-```yaml title="conf/config.yaml"
+```yaml
plugin_attr:
- prometheus:
- export_uri: /apisix/metrics
+ prometheus: # Plugin: prometheus attributes
+ export_uri: /apisix/prometheus/metrics # Set the URI for the Prometheus metrics endpoint.
+ metric_prefix: apisix_ # Set the prefix for Prometheus metrics generated by APISIX.
+ enable_export_server: true # Enable the Prometheus export server.
+ export_addr: # Set the address for the Prometheus export server.
+ ip: 127.0.0.1 # Set the IP.
+ port: 9091 # Set the port.
+ # metrics: # Create extra labels for metrics.
+ # http_status: # These metrics will be prefixed with `apisix_`.
+ # extra_labels: # Set the extra labels for http_status metrics.
+ # - upstream_addr: $upstream_addr
+ # - status: $upstream_status
+ # expire: 0 # The expiration time of metrics in seconds.
+ # 0 means the metrics will not expire.
+ # http_latency:
+ # extra_labels: # Set the extra labels for http_latency metrics.
+ # - upstream_addr: $upstream_addr
+ # expire: 0 # The expiration time of metrics in seconds.
+ # 0 means the metrics will not expire.
+ # bandwidth:
+ # extra_labels: # Set the extra labels for bandwidth metrics.
+ # - upstream_addr: $upstream_addr
+ # expire: 0 # The expiration time of metrics in seconds.
+ # 0 means the metrics will not expire.
+ # default_buckets: # Set the default buckets for the `http_latency` metrics histogram.
+ # - 10
+ # - 50
+ # - 100
+ # - 200
+ # - 500
+ # - 1000
+ # - 2000
+ # - 5000
+ # - 10000
+ # - 30000
+ # - 60000
+ # - 500
```
-### Specifying `metrics`
+You can use the [Nginx variable](https://nginx.org/en/docs/http/ngx_http_core_module.html) to create `extra_labels`. See [add extra labels](/hub/prometheus#add-extra-labels-for-metrics).
-For http request related metrics, you could specify extra labels, which match the APISIX variables.
+Reload APISIX for changes to take effect.
-If you specify label for nonexist APISIX variable, the label value would be "".
+## Attribute
-Currently, only below metrics are supported:
+| Name | Type | Required | Default | Valid values | Description |
+| ------------- | ------- | -------- | ------- | ------------ | ------------------------------------------ |
+| `prefer_name` | boolean | | False | | If true, export Route/Service name instead of their ID in Prometheus metrics. |
-* http_status
-* http_latency
-* bandwidth
+## Metrics
-Here is a configuration example:
+There are different types of metrics in Prometheus. To understand their differences, see [metrics types](https://prometheus.io/docs/concepts/metric_types/).
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- metrics:
- http_status:
- extra_labels:
- - upstream_addr: $upstream_addr
- - upstream_status: $upstream_status
-```
+The following metrics are exported by the `prometheus` Plugin by default. See [get APISIX metrics](#get-apisix-metrics) for an example. Note that some metrics, such as `apisix_batch_process_entries`, are not readily visible if there are no data.
-### Specifying `default_buckets`
+| Name | Type | Description |
+| ------------------------------ | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| apisix_bandwidth | counter | Total amount of traffic flowing through APISIX in bytes. |
+| apisix_etcd_modify_indexes | gauge | Number of changes to etcd by APISIX keys. |
+| apisix_batch_process_entries | gauge | Number of remaining entries in a batch when sending data in batches, such as with `http logger`, and other logging Plugins. |
+| apisix_etcd_reachable | gauge | Whether APISIX can reach etcd. A value of `1` represents reachable and `0` represents unreachable. |
+| apisix_http_status | counter | HTTP status codes returned from upstream Services. |
+| apisix_http_requests_total | gauge | Number of HTTP requests from clients. |
+| apisix_nginx_http_current_connections | gauge | Number of current connections with clients. |
+| apisix_nginx_metric_errors_total | counter | Total number of `nginx-lua-prometheus` errors. |
+| apisix_http_latency | histogram | HTTP request latency in milliseconds. |
+| apisix_node_info | gauge | Information of the APISIX node, such as host name. |
+| apisix_shared_dict_capacity_bytes | gauge | The total capacity of an [NGINX shared dictionary](https://github.com/openresty/lua-nginx-module#ngxshareddict). |
+| apisix_shared_dict_free_space_bytes | gauge | The remaining space in an [NGINX shared dictionary](https://github.com/openresty/lua-nginx-module#ngxshareddict). |
+| apisix_upstream_status | gauge | Health check status of upstream nodes, available if health checks are configured on the upstream. A value of `1` represents healthy and `0` represents unhealthy. |
+| apisix_stream_connection_total | counter | Total number of connections handled per stream Route. |
-`DEFAULT_BUCKETS` is the default value for bucket array in `http_latency` metrics.
+## Labels
-You can change the `DEFAULT_BUCKETS` by configuring `default_buckets` attribute in you configuration file.
+[Labels](https://prometheus.io/docs/practices/naming/#labels) are attributes of metrics that are used to differentiate metrics.
-Here is a configuration example:
+For example, the `apisix_http_status` metric can be labeled with `route` information to identify which Route the HTTP status originates from.
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- default_buckets:
- - 15
- - 55
- - 105
- - 205
- - 505
-```
+The following are labels for a non-exhaustive list of APISIX metrics and their descriptions.
-### Specifying `expire`
+### Labels for `apisix_http_status`
-`expire` sets the expiration time of `apisix_http_status`, `apisix_bandwidth`, and `apisix_http_latency` metrics in seconds. When set to 0, metrics will not expire.
+The following labels are used to differentiate `apisix_http_status` metrics.
-Here is a configuration example:
+| Name | Description |
+| ------------ | ----------------------------------------------------------------------------------------------------------------------------- |
+| code | HTTP response code returned by the upstream node. |
+| route | ID of the Route that the HTTP status originates from when `prefer_name` is `false` (default), and name of the Route when `prefer_name` to `true`. Default to an empty string if a request does not match any Route. |
+| route_id | Available only in Enterprise. ID of the Route that the HTTP status originates from regardless of the `prefer_name` setting. |
+| matched_uri | URI of the Route that matches the request. Default to an empty string if a request does not match any Route. |
+| matched_host | Host of the Route that matches the request. Default to an empty string if a request does not match any Route, or host is not configured on the Route. |
+| service | ID of the Service that the HTTP status originates from when `prefer_name` is `false` (default), and name of the Service when `prefer_name` to `true`. Default to the configured value of host on the Route if the matched Route does not belong to any Service. |
+| service_id | Available only in Enterprise. ID of the Service that the HTTP status originates from regardless of the `prefer_name` setting. |
+| consumer | Name of the Consumer associated with a request. Default to an empty string if no Consumer is associated with the request. |
+| node | IP address of the upstream node. |
+| gateway_group_id | Available only in Enterprise. ID of the gateway group that the HTTP status originates from. |
+| instance_id | Available only in Enterprise. ID of the gateway instance that the HTTP status originates from. |
+| api_product_id | Available only in Enterprise. Product ID that the HTTP status originates from. |
Review Comment:
Remove all the labels that are `Available only in Enterprise.` There are 15 of them.
##########
docs/en/latest/plugins/prometheus.md:
##########
@@ -27,439 +27,471 @@ description: This document contains information about the Apache APISIX promethe
#
-->
-## Description
-
-The `prometheus` Plugin exports metrics in [Prometheus exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/#exposition-formats).
+<head>
+ <link rel="canonical" href="https://docs.api7.ai/hub/prometheus" />
+</head>
-## Attributes
+## Description
-| Name | Type | Required | Default | Description |
-| ----------- | ------- | -------- | ------- | --------------------------------------------------------------------------------- |
-| prefer_name | boolean | False | false | When set to `true`, prints Route/Service name instead of ID in Prometheus metric. |
+The `prometheus` Plugin provides the capability to integrate APISIX with [Prometheus](https://prometheus.io).
-### Specifying `export_uri`
+After enabling the Plugin, APISIX will start collecting relevant metrics, such as API requests and latencies, and exporting them in a [text-based exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/#exposition-formats) to Prometheus. You can then create event monitoring and alerting in Prometheus to monitor the health of your API gateway and APIs.
-You can change the default export URI by configuring the `export_uri` attribute under `plugin_attr` in your configuration file (`conf/config.yaml`).
+## Static Configurations
-| Name | Type | Default | Description |
-| ---------- | ------ | ---------------------------- | ------------------------------------- |
-| export_uri | string | "/apisix/prometheus/metrics" | URI to export the Prometheus metrics. |
+By default, `prometheus` configurations are pre-configured in the [default configuration](https://github.com/apache/apisix/blob/master/apisix/cli/config.lua).
-Here is a configuration example:
+To customize these values, add the corresponding configurations to `config.yaml`. For example:
-```yaml title="conf/config.yaml"
+```yaml
plugin_attr:
- prometheus:
- export_uri: /apisix/metrics
+ prometheus: # Plugin: prometheus attributes
+ export_uri: /apisix/prometheus/metrics # Set the URI for the Prometheus metrics endpoint.
+ metric_prefix: apisix_ # Set the prefix for Prometheus metrics generated by APISIX.
+ enable_export_server: true # Enable the Prometheus export server.
+ export_addr: # Set the address for the Prometheus export server.
+ ip: 127.0.0.1 # Set the IP.
+ port: 9091 # Set the port.
+ # metrics: # Create extra labels for metrics.
+ # http_status: # These metrics will be prefixed with `apisix_`.
+ # extra_labels: # Set the extra labels for http_status metrics.
+ # - upstream_addr: $upstream_addr
+ # - status: $upstream_status
+ # expire: 0 # The expiration time of metrics in seconds.
+ # 0 means the metrics will not expire.
+ # http_latency:
+ # extra_labels: # Set the extra labels for http_latency metrics.
+ # - upstream_addr: $upstream_addr
+ # expire: 0 # The expiration time of metrics in seconds.
+ # 0 means the metrics will not expire.
+ # bandwidth:
+ # extra_labels: # Set the extra labels for bandwidth metrics.
+ # - upstream_addr: $upstream_addr
+ # expire: 0 # The expiration time of metrics in seconds.
+ # 0 means the metrics will not expire.
+ # default_buckets: # Set the default buckets for the `http_latency` metrics histogram.
+ # - 10
+ # - 50
+ # - 100
+ # - 200
+ # - 500
+ # - 1000
+ # - 2000
+ # - 5000
+ # - 10000
+ # - 30000
+ # - 60000
+ # - 500
```
-### Specifying `metrics`
+You can use the [Nginx variable](https://nginx.org/en/docs/http/ngx_http_core_module.html) to create `extra_labels`. See [add extra labels](/hub/prometheus#add-extra-labels-for-metrics).
-For http request related metrics, you could specify extra labels, which match the APISIX variables.
+Reload APISIX for changes to take effect.
-If you specify label for nonexist APISIX variable, the label value would be "".
+## Attribute
-Currently, only below metrics are supported:
+| Name | Type | Required | Default | Valid values | Description |
+| ------------- | ------- | -------- | ------- | ------------ | ------------------------------------------ |
+| `prefer_name` | boolean | | False | | If true, export Route/Service name instead of their ID in Prometheus metrics. |
-* http_status
-* http_latency
-* bandwidth
+## Metrics
-Here is a configuration example:
+There are different types of metrics in Prometheus. To understand their differences, see [metrics types](https://prometheus.io/docs/concepts/metric_types/).
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- metrics:
- http_status:
- extra_labels:
- - upstream_addr: $upstream_addr
- - upstream_status: $upstream_status
-```
+The following metrics are exported by the `prometheus` Plugin by default. See [get APISIX metrics](#get-apisix-metrics) for an example. Note that some metrics, such as `apisix_batch_process_entries`, are not readily visible if there are no data.
-### Specifying `default_buckets`
+| Name | Type | Description |
+| ------------------------------ | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| apisix_bandwidth | counter | Total amount of traffic flowing through APISIX in bytes. |
+| apisix_etcd_modify_indexes | gauge | Number of changes to etcd by APISIX keys. |
+| apisix_batch_process_entries | gauge | Number of remaining entries in a batch when sending data in batches, such as with `http logger`, and other logging Plugins. |
+| apisix_etcd_reachable | gauge | Whether APISIX can reach etcd. A value of `1` represents reachable and `0` represents unreachable. |
+| apisix_http_status | counter | HTTP status codes returned from upstream Services. |
+| apisix_http_requests_total | gauge | Number of HTTP requests from clients. |
+| apisix_nginx_http_current_connections | gauge | Number of current connections with clients. |
+| apisix_nginx_metric_errors_total | counter | Total number of `nginx-lua-prometheus` errors. |
+| apisix_http_latency | histogram | HTTP request latency in milliseconds. |
+| apisix_node_info | gauge | Information of the APISIX node, such as host name. |
+| apisix_shared_dict_capacity_bytes | gauge | The total capacity of an [NGINX shared dictionary](https://github.com/openresty/lua-nginx-module#ngxshareddict). |
+| apisix_shared_dict_free_space_bytes | gauge | The remaining space in an [NGINX shared dictionary](https://github.com/openresty/lua-nginx-module#ngxshareddict). |
+| apisix_upstream_status | gauge | Health check status of upstream nodes, available if health checks are configured on the upstream. A value of `1` represents healthy and `0` represents unhealthy. |
+| apisix_stream_connection_total | counter | Total number of connections handled per stream Route. |
-`DEFAULT_BUCKETS` is the default value for bucket array in `http_latency` metrics.
+## Labels
-You can change the `DEFAULT_BUCKETS` by configuring `default_buckets` attribute in you configuration file.
+[Labels](https://prometheus.io/docs/practices/naming/#labels) are attributes of metrics that are used to differentiate metrics.
-Here is a configuration example:
+For example, the `apisix_http_status` metric can be labeled with `route` information to identify which Route the HTTP status originates from.
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- default_buckets:
- - 15
- - 55
- - 105
- - 205
- - 505
-```
+The following are labels for a non-exhaustive list of APISIX metrics and their descriptions.
-### Specifying `expire`
+### Labels for `apisix_http_status`
-`expire` sets the expiration time of `apisix_http_status`, `apisix_bandwidth`, and `apisix_http_latency` metrics in seconds. When set to 0, metrics will not expire.
+The following labels are used to differentiate `apisix_http_status` metrics.
-Here is a configuration example:
+| Name | Description |
+| ------------ | ----------------------------------------------------------------------------------------------------------------------------- |
+| code | HTTP response code returned by the upstream node. |
+| route | ID of the Route that the HTTP status originates from when `prefer_name` is `false` (default), and name of the Route when `prefer_name` to `true`. Default to an empty string if a request does not match any Route. |
+| route_id | Available only in Enterprise. ID of the Route that the HTTP status originates from regardless of the `prefer_name` setting. |
+| matched_uri | URI of the Route that matches the request. Default to an empty string if a request does not match any Route. |
+| matched_host | Host of the Route that matches the request. Default to an empty string if a request does not match any Route, or host is not configured on the Route. |
+| service | ID of the Service that the HTTP status originates from when `prefer_name` is `false` (default), and name of the Service when `prefer_name` to `true`. Default to the configured value of host on the Route if the matched Route does not belong to any Service. |
+| service_id | Available only in Enterprise. ID of the Service that the HTTP status originates from regardless of the `prefer_name` setting. |
+| consumer | Name of the Consumer associated with a request. Default to an empty string if no Consumer is associated with the request. |
+| node | IP address of the upstream node. |
+| gateway_group_id | Available only in Enterprise. ID of the gateway group that the HTTP status originates from. |
+| instance_id | Available only in Enterprise. ID of the gateway instance that the HTTP status originates from. |
+| api_product_id | Available only in Enterprise. Product ID that the HTTP status originates from. |
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- expire: 86400
-```
+### Labels for `apisix_bandwidth`
-## Metrics endpoint
+The following labels are used to differentiate `apisix_bandwidth` metrics.
-This Plugin will add the metrics endpoint `/apisix/prometheus/metrics` or your custom export URI for exposing the metrics.
+| Name | Description |
+| ---------- | ----------------------------------------------------------------------------------------------------------------------------- |
+| type | Type of traffic, `egress` or `ingress`. |
+| route | ID of the Route that bandwidth corresponds to when `prefer_name` is `false` (default), and name of the Route when `prefer_name` to `true`. Default to an empty string if a request does not match any Route. |
+| route_id | Available only in Enterprise. ID of the Route that bandwidth corresponds to regardless of the `prefer_name` setting. |
+| service | ID of the Service that bandwidth corresponds to when `prefer_name` is `false` (default), and name of the Service when `prefer_name` to `true`. Default to the configured value of host on the Route if the matched Route does not belong to any Service. |
+| service_id | Available only in Enterprise. ID of the Service that bandwidth corresponds to regardless of the `prefer_name` setting. |
+| consumer | Name of the Consumer associated with a request. Default to an empty string if no Consumer is associated with the request. |
+| node | IP address of the upstream node. |
+| gateway_group_id | Available only in Enterprise. ID of the gateway group that bandwidth corresponds to. |
+| instance_id | Available only in Enterprise. ID of the gateway instance that bandwidth corresponds to. |
+| api_product_id | Available only in Enterprise. Product ID that bandwidth corresponds to. |
-These metrics are exposed by a separate Prometheus server address. By default, the address is `127.0.0.1:9091`. You can change it in your configuration file (`conf/config.yaml`):
+### Labels for `apisix_http_latency`
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- export_addr:
- ip: ${{INTRANET_IP}}
- port: 9092
-```
+The following labels are used to differentiate `apisix_http_latency` metrics.
-Now, if the environment variable `INTRANET_IP` is `172.1.1.1`, APISIX will export the metrics via `172.1.1.1:9092`.
+| Name | Description |
+| ---------- | ----------------------------------------------------------------------------------------------------------------------------------- |
+| type | Type of latencies. See [latency types](#latency-types) for details. |
+| route | ID of the Route that latencies correspond to when `prefer_name` is `false` (default), and name of the Route when `prefer_name` to `true`. Default to an empty string if a request does not match any Route. |
+| route_id | Available only in Enterprise. ID of the Route that latencies correspond to regardless of the `prefer_name` setting. |
+| service | ID of the Service that latencies correspond to when `prefer_name` is `false` (default), and name of the Service when `prefer_name` to `true`. Default to the configured value of host on the Route if the matched Route does not belong to any Service. |
+| service_id | Available only in Enterprise. ID of the Service that latencies correspond to regardless of the `prefer_name` setting. |
+| consumer | Name of the Consumer associated with latencies. Default to an empty string if no Consumer is associated with the request. |
+| node | IP address of the upstream node associated with latencies. |
+| gateway_group_id | Available only in Enterprise. ID of the gateway group that latencies correspond to. |
+| instance_id | Available only in Enterprise. ID of the gateway instance that latencies correspond to. |
+| api_product_id | Available only in Enterprise. Product ID that latencies correspond to. |
-If you still want to expose the metrics via the data plane port (default: `9080`), you can configure it as shown below:
+#### Latency Types
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- enable_export_server: false
+`apisix_http_latency` can be labeled with one of the three types:
+
+* `request` represents the time elapsed between the first byte was read from the client and the log write after the last byte was sent to the client.
+
+* `upstream` represents the time elapsed waiting on responses from the upstream Service.
+
+* `apisix` represents the difference between the `request` latency and `upstream` latency.
+
+In other words, the APISIX latency is not only attributed to the Lua processing. It should be understood as follows:
+
+```text
+APISIX latency
+ = downstream request time - upstream response time
+ = downstream traffic latency + NGINX latency
```
-You can then expose it by using the [public-api](public-api.md) Plugin.
+### Labels for `apisix_upstream_status`
-:::info IMPORTANT
+The following labels are used to differentiate `apisix_upstream_status` metrics.
-If the Prometheus plugin collects too many metrics, it will take CPU resources to calculate the metric data when getting the metrics via URI, which may affect APISIX to process normal requests. To solve this problem, APISIX exposes the URI and calculates the metrics in the [privileged agent](https://github.com/openresty/lua-resty-core/blob/master/lib/ngx/process.md#enable_privileged_agent).
-If the URI is exposed using the public-api plugin, then APISIX will calculate the metric data in a normal worker process, which may still affect APISIX processing of normal requests.
+| Name | Description |
+| ---------- | --------------------------------------------------------------------------------------------------- |
+| name | Resource ID corresponding to the upstream configured with health checks, such as `/apisix/routes/1` and `/apisix/upstreams/1`. |
+| ip | IP address of the upstream node. |
+| port | Port number of the node. |
-This feature requires APISIX to run on [APISIX-Runtime](../FAQ.md#how-do-i-build-the-apisix-runtime-environment).
+## Examples
-:::
+The examples below demonstrate how you can work with the `prometheus` Plugin for different scenarios.
-## Enable Plugin
+### Get APISIX Metrics
-The `prometheus` Plugin can be enabled with an empty table.
+The following example demonstrates how you can get metrics from APISIX.
-The example below shows how you can configure the Plugin on a specific Route:
+The default Prometheus metrics endpoint and other Prometheus related configurations can be found in the [static configuration](#static-configurations). If you would like to customize these configuration, see [configuration based on environments](https://apisix.apache.org/docs/apisix/profile/).
-:::note
-You can fetch the `admin_key` from `config.yaml` and save to an environment variable with the following command:
+If you deploy APISIX in a containerized environment and would like to access the Prometheus metrics endpoint externally, update the configuration file as follows and reload APISIX:
-```bash
-admin_key=$(yq '.deployment.admin.admin_key[0].key' conf/config.yaml | sed 's/"//g')
+```yaml title="conf/config.yaml"
+plugin_attr:
+ prometheus:
+ export_addr:
+ ip: 0.0.0.0
```
-:::
+Send a request to the APISIX Prometheus metrics endpoint:
```shell
-curl http://127.0.0.1:9180/apisix/admin/routes/1 -H "X-API-KEY: $admin_key" -X PUT -d '
-{
- "uri": "/hello",
- "plugins": {
- "prometheus":{}
- },
- "upstream": {
- "type": "roundrobin",
- "nodes": {
- "127.0.0.1:80": 1
- }
- }
-}'
+curl "http://127.0.0.1:9091/apisix/prometheus/metrics"
```
-:::note
-
-When `prefer_name` is set to `true` make sure to not duplicate names for multiple Routes/Services or it could be misleading.
-
-:::
+You should see an output similar to the following:
-<!-- You can use [APISIX Dashboard](https://github.com/apache/apisix-dashboard) to complete the above operations through the web console.
-
-First, add a Route:
+```text
+# HELP apisix_bandwidth Total bandwidth in bytes consumed per Service in Apisix
+# TYPE apisix_bandwidth counter
+apisix_bandwidth{type="egress",route="",service="",consumer="",node=""} 8417
+apisix_bandwidth{type="egress",route="1",service="",consumer="",node="127.0.0.1"} 1420
+apisix_bandwidth{type="egress",route="2",service="",consumer="",node="127.0.0.1"} 1420
+apisix_bandwidth{type="ingress",route="",service="",consumer="",node=""} 189
+apisix_bandwidth{type="ingress",route="1",service="",consumer="",node="127.0.0.1"} 332
+apisix_bandwidth{type="ingress",route="2",service="",consumer="",node="127.0.0.1"} 332
+# HELP apisix_etcd_modify_indexes Etcd modify index for APISIX keys
+# TYPE apisix_etcd_modify_indexes gauge
+apisix_etcd_modify_indexes{key="consumers"} 0
+apisix_etcd_modify_indexes{key="global_rules"} 0
+...
+```
-
+### Expose APISIX Metrics on Public API Endpoint
-Then add prometheus plugin:
+The following example demonstrates how you can disable the Prometheus export server that, by default, exposes an endpoint on port `9091`, and expose APISIX Prometheus metrics on a new public API endpoint on port `9080`, which APISIX uses to listen to other client requests.
- -->
+Disable the Prometheus export server in the configuration file and reload APISIX for changes to take effect:
-## Fetching metrics
+```yaml title="conf/config.yaml"
+plugin_attr:
+ prometheus:
+ enable_export_server: false
+```
-You can fetch the metrics from the specified export URI (default: `/apisix/prometheus/metrics`):
+Next, create a Route with [`public-api`](./public-api.md) Plugin and expose a public API endpoint for APISIX metrics:
```shell
-curl -i http://127.0.0.1:9091/apisix/prometheus/metrics
+curl "http://127.0.0.1:9180/apisix/admin/routes/prometheus-metrics" -X PUT \
+ -H "X-API-KEY: ${admin_key}" \
+ -d '{
+ "uri": "/apisix/prometheus/metrics",
+ "plugins": {
+ "public-api": {}
+ }
+ }'
```
-You can add this address to Prometheus to fetch the data:
+Send a request to the new metrics endpoint to verify:
-```yaml
-scrape_configs:
- - job_name: "apisix"
- scrape_interval: 15s # This value will be related to the time range of the rate function in Prometheus QL. The time range in the rate function should be at least twice this value.
- metrics_path: "/apisix/prometheus/metrics"
- static_configs:
- - targets: ["127.0.0.1:9091"]
+```shell
+curl "http://127.0.0.1:9080/apisix/prometheus/metrics"
```
-Now, you will be able to check the status in your Prometheus console:
-
-
+You should see an output similar to the following:
-
-
-## Using Grafana to graph the metrics
-
-Metrics exported by the `prometheus` Plugin can be graphed in Grafana using a drop in dashboard.
-
-To set it up, download [Grafana dashboard meta](https://github.com/apache/apisix/blob/master/docs/assets/other/json/apisix-grafana-dashboard.json) and import it in Grafana. Or, you can go to [Grafana official](https://grafana.com/grafana/dashboards/11719) for Grafana metadata.
+```text
+# HELP apisix_http_requests_total The total number of client requests since APISIX started
+# TYPE apisix_http_requests_total gauge
+apisix_http_requests_total 1
+# HELP apisix_nginx_http_current_connections Number of HTTP connections
+# TYPE apisix_nginx_http_current_connections gauge
+apisix_nginx_http_current_connections{state="accepted"} 1
+apisix_nginx_http_current_connections{state="active"} 1
+apisix_nginx_http_current_connections{state="handled"} 1
+apisix_nginx_http_current_connections{state="reading"} 0
+apisix_nginx_http_current_connections{state="waiting"} 0
+apisix_nginx_http_current_connections{state="writing"} 1
+...
+```
-
+### Integrate APISIX with Prometheus and Grafana
-
+To learn about how to collect APISIX metrics with Prometheus and visualize them in Grafana, see [how-to guide](/apisix/how-to-guide/observability/monitor-apisix-with-prometheus).
-
+### Monitor Upstream Health Statuses
-
+The following example demonstrates how to monitor the health status of upstream nodes.
-## Available HTTP metrics
+Create a Route with the `prometheus` Plugin and configure upstream active health checks:
-The following metrics are exported by the `prometheus` Plugin:
+```shell
+curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
+ -H "X-API-KEY: ${admin_key}" \
+ -d '{
+ "id": "prometheus-route",
+ "uri": "/get",
+ "plugins": {
+ "prometheus": {}
+ },
+ "upstream": {
+ "type": "roundrobin",
+ "nodes": {
+ "httpbin.org:80": 1,
+ "127.0.0.1:20001": 1
+ },
+ "checks": {
+ "active": {
+ "timeout": 5,
+ "http_path": "/status",
+ "healthy": {
+ "interval": 2,
+ "successes": 1
+ },
+ "unhealthy": {
+ "interval": 1,
+ "http_failures": 2
+ }
+ },
+ "passive": {
+ "healthy": {
+ "http_statuses": [200, 201],
+ "successes": 3
+ },
+ "unhealthy": {
+ "http_statuses": [500],
+ "http_failures": 3,
+ "tcp_failures": 3
+ }
+ }
+ }
+ }
+ }'
+```
-- Status code: HTTP status code returned from Upstream services. They are available for a single service and across all services.
+Send a request to the APISIX Prometheus metrics endpoint:
- The available attributes are:
+```shell
+curl "http://127.0.0.1:9091/apisix/prometheus/metrics"
+```
- | Name | Description |
- |--------------|-------------------------------------------------------------------------------------------------------------------------------|
- | code | HTTP status code returned by the upstream service. |
- | route | `route_id` of the matched Route with request. Defaults to an empty string if the Routes don't match. |
- | matched_uri | `uri` of the Route matching the request. Defaults to an empty string if the Routes don't match. |
- | matched_host | `host` of the Route matching the request. Defaults to an empty string if the Routes don't match. |
- | service | `service_id` of the Route matching the request. If the Route does not have a `service_id` configured, it defaults to `$host`. |
- | consumer | `consumer_name` of the Consumer matching the request. Defaults to an empty string if it does not match. |
- | node | IP address of the Upstream node. |
+You should see an output similar to the following:
-- Bandwidth: Total amount of traffic (ingress and egress) flowing through APISIX. Total bandwidth of a service can also be obtained.
+```text
+# HELP apisix_upstream_status upstream status from health check
+# TYPE apisix_upstream_status gauge
+apisix_upstream_status{name="/apisix/routes/1",ip="54.237.103.220",port="80"} 1
+apisix_upstream_status{name="/apisix/routes/1",ip="127.0.0.1",port="20001"} 0
+```
- The available attributes are:
+This shows that the upstream node `httpbin.org:80` is healthy and the upstream node `127.0.0.1:20001` is unhealthy.
- | Name | Description |
- |----------|-------------------------------------------------------------------------------------------------------------------------------|
- | type | Type of traffic (egress/ingress). |
- | route | `route_id` of the matched Route with request. Defaults to an empty string if the Routes don't match. |
- | service | `service_id` of the Route matching the request. If the Route does not have a `service_id` configured, it defaults to `$host`. |
- | consumer | `consumer_name` of the Consumer matching the request. Defaults to an empty string if it does not match. |
- | node | IP address of the Upstream node. |
+To learn more about how to configure active and passive health checks, see [health checks](/docs/apisix/tutorials/health-check/).
-- etcd reachability: A gauge type representing whether etcd can be reached by APISIX. A value of `1` represents reachable and `0` represents unreachable.
-- Connections: Nginx connection metrics like active, reading, writing, and number of accepted connections.
-- Batch process entries: A gauge type useful when Plugins like [syslog](./syslog.md), [http-logger](./http-logger.md), [tcp-logger](./tcp-logger.md), [udp-logger](./udp-logger.md), and [zipkin](./zipkin.md) use batch process to send data. Entries that hasn't been sent in batch process will be counted in the metrics.
-- Latency: Histogram of the request time per service in different dimensions.
+### Add Extra Labels for Metrics
- The available attributes are:
+The following example demonstrates how to add additional labels to metrics and use the [Nginx variable](https://nginx.org/en/docs/http/ngx_http_core_module.html) in label values.
- | Name | Description |
- |----------|-------------------------------------------------------------------------------------------------------------------------------------|
- | type | Value can be one of `apisix`, `upstream`, or `request`. This translates to latency caused by APISIX, Upstream, or both (their sum). |
- | route | `route_id` of the matched Route with request. Defaults to an empty string if the Routes don't match. |
- | service | `service_id` of the Route matching the request. If the Route does not have a `service_id` configured, it defaults to `$host`. |
- | consumer | `consumer_name` of the Consumer matching the request. Defaults to an empty string if it does not match. |
- | node | IP address of the Upstream node. |
+Currently, only the following metrics support extra labels:
-- Info: Information about the APISIX node.
-- Shared dict: The capacity and free space of all nginx.shared.DICT in APISIX.
+* apisix_http_status
+* apisix_http_latency
+* apisix_bandwidth
-- `apisix_upstream_status`: Health check result status of upstream nodes. A value of `1` represents healthy and `0` represents unhealthy.
+Include the following configurations in the [configuration based on environments](https://apisix.apache.org/docs/apisix/profile/) to add labels for metrics and reload APISIX for changes to take effect:
Review Comment:
Wrong link.
```suggestion
Include the following configurations in the configuration file to add labels for metrics and reload APISIX for changes to take effect:
```
This should be sufficient because:
<img width="327" alt="image" src="https://github.com/user-attachments/assets/450fde33-00ad-42dd-8ff6-a97718cd0ecb" />
##########
docs/en/latest/plugins/prometheus.md:
##########
@@ -27,439 +27,471 @@ description: This document contains information about the Apache APISIX promethe
#
-->
-## Description
-
-The `prometheus` Plugin exports metrics in [Prometheus exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/#exposition-formats).
+<head>
+ <link rel="canonical" href="https://docs.api7.ai/hub/prometheus" />
+</head>
-## Attributes
+## Description
-| Name | Type | Required | Default | Description |
-| ----------- | ------- | -------- | ------- | --------------------------------------------------------------------------------- |
-| prefer_name | boolean | False | false | When set to `true`, prints Route/Service name instead of ID in Prometheus metric. |
+The `prometheus` Plugin provides the capability to integrate APISIX with [Prometheus](https://prometheus.io).
-### Specifying `export_uri`
+After enabling the Plugin, APISIX will start collecting relevant metrics, such as API requests and latencies, and exporting them in a [text-based exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/#exposition-formats) to Prometheus. You can then create event monitoring and alerting in Prometheus to monitor the health of your API gateway and APIs.
-You can change the default export URI by configuring the `export_uri` attribute under `plugin_attr` in your configuration file (`conf/config.yaml`).
+## Static Configurations
-| Name | Type | Default | Description |
-| ---------- | ------ | ---------------------------- | ------------------------------------- |
-| export_uri | string | "/apisix/prometheus/metrics" | URI to export the Prometheus metrics. |
+By default, `prometheus` configurations are pre-configured in the [default configuration](https://github.com/apache/apisix/blob/master/apisix/cli/config.lua).
-Here is a configuration example:
+To customize these values, add the corresponding configurations to `config.yaml`. For example:
-```yaml title="conf/config.yaml"
+```yaml
plugin_attr:
- prometheus:
- export_uri: /apisix/metrics
+ prometheus: # Plugin: prometheus attributes
+ export_uri: /apisix/prometheus/metrics # Set the URI for the Prometheus metrics endpoint.
+ metric_prefix: apisix_ # Set the prefix for Prometheus metrics generated by APISIX.
+ enable_export_server: true # Enable the Prometheus export server.
+ export_addr: # Set the address for the Prometheus export server.
+ ip: 127.0.0.1 # Set the IP.
+ port: 9091 # Set the port.
+ # metrics: # Create extra labels for metrics.
+ # http_status: # These metrics will be prefixed with `apisix_`.
+ # extra_labels: # Set the extra labels for http_status metrics.
+ # - upstream_addr: $upstream_addr
+ # - status: $upstream_status
+ # expire: 0 # The expiration time of metrics in seconds.
+ # 0 means the metrics will not expire.
+ # http_latency:
+ # extra_labels: # Set the extra labels for http_latency metrics.
+ # - upstream_addr: $upstream_addr
+ # expire: 0 # The expiration time of metrics in seconds.
+ # 0 means the metrics will not expire.
+ # bandwidth:
+ # extra_labels: # Set the extra labels for bandwidth metrics.
+ # - upstream_addr: $upstream_addr
+ # expire: 0 # The expiration time of metrics in seconds.
+ # 0 means the metrics will not expire.
+ # default_buckets: # Set the default buckets for the `http_latency` metrics histogram.
+ # - 10
+ # - 50
+ # - 100
+ # - 200
+ # - 500
+ # - 1000
+ # - 2000
+ # - 5000
+ # - 10000
+ # - 30000
+ # - 60000
+ # - 500
```
-### Specifying `metrics`
+You can use the [Nginx variable](https://nginx.org/en/docs/http/ngx_http_core_module.html) to create `extra_labels`. See [add extra labels](/hub/prometheus#add-extra-labels-for-metrics).
-For http request related metrics, you could specify extra labels, which match the APISIX variables.
+Reload APISIX for changes to take effect.
-If you specify label for nonexist APISIX variable, the label value would be "".
+## Attribute
-Currently, only below metrics are supported:
+| Name | Type | Required | Default | Valid values | Description |
+| ------------- | ------- | -------- | ------- | ------------ | ------------------------------------------ |
+| `prefer_name` | boolean | | False | | If true, export Route/Service name instead of their ID in Prometheus metrics. |
-* http_status
-* http_latency
-* bandwidth
+## Metrics
-Here is a configuration example:
+There are different types of metrics in Prometheus. To understand their differences, see [metrics types](https://prometheus.io/docs/concepts/metric_types/).
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- metrics:
- http_status:
- extra_labels:
- - upstream_addr: $upstream_addr
- - upstream_status: $upstream_status
-```
+The following metrics are exported by the `prometheus` Plugin by default. See [get APISIX metrics](#get-apisix-metrics) for an example. Note that some metrics, such as `apisix_batch_process_entries`, are not readily visible if there are no data.
-### Specifying `default_buckets`
+| Name | Type | Description |
+| ------------------------------ | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| apisix_bandwidth | counter | Total amount of traffic flowing through APISIX in bytes. |
+| apisix_etcd_modify_indexes | gauge | Number of changes to etcd by APISIX keys. |
+| apisix_batch_process_entries | gauge | Number of remaining entries in a batch when sending data in batches, such as with `http logger`, and other logging Plugins. |
+| apisix_etcd_reachable | gauge | Whether APISIX can reach etcd. A value of `1` represents reachable and `0` represents unreachable. |
+| apisix_http_status | counter | HTTP status codes returned from upstream Services. |
+| apisix_http_requests_total | gauge | Number of HTTP requests from clients. |
+| apisix_nginx_http_current_connections | gauge | Number of current connections with clients. |
+| apisix_nginx_metric_errors_total | counter | Total number of `nginx-lua-prometheus` errors. |
+| apisix_http_latency | histogram | HTTP request latency in milliseconds. |
+| apisix_node_info | gauge | Information of the APISIX node, such as host name. |
+| apisix_shared_dict_capacity_bytes | gauge | The total capacity of an [NGINX shared dictionary](https://github.com/openresty/lua-nginx-module#ngxshareddict). |
+| apisix_shared_dict_free_space_bytes | gauge | The remaining space in an [NGINX shared dictionary](https://github.com/openresty/lua-nginx-module#ngxshareddict). |
+| apisix_upstream_status | gauge | Health check status of upstream nodes, available if health checks are configured on the upstream. A value of `1` represents healthy and `0` represents unhealthy. |
+| apisix_stream_connection_total | counter | Total number of connections handled per stream Route. |
-`DEFAULT_BUCKETS` is the default value for bucket array in `http_latency` metrics.
+## Labels
-You can change the `DEFAULT_BUCKETS` by configuring `default_buckets` attribute in you configuration file.
+[Labels](https://prometheus.io/docs/practices/naming/#labels) are attributes of metrics that are used to differentiate metrics.
-Here is a configuration example:
+For example, the `apisix_http_status` metric can be labeled with `route` information to identify which Route the HTTP status originates from.
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- default_buckets:
- - 15
- - 55
- - 105
- - 205
- - 505
-```
+The following are labels for a non-exhaustive list of APISIX metrics and their descriptions.
-### Specifying `expire`
+### Labels for `apisix_http_status`
-`expire` sets the expiration time of `apisix_http_status`, `apisix_bandwidth`, and `apisix_http_latency` metrics in seconds. When set to 0, metrics will not expire.
+The following labels are used to differentiate `apisix_http_status` metrics.
-Here is a configuration example:
+| Name | Description |
+| ------------ | ----------------------------------------------------------------------------------------------------------------------------- |
+| code | HTTP response code returned by the upstream node. |
+| route | ID of the Route that the HTTP status originates from when `prefer_name` is `false` (default), and name of the Route when `prefer_name` to `true`. Default to an empty string if a request does not match any Route. |
+| route_id | Available only in Enterprise. ID of the Route that the HTTP status originates from regardless of the `prefer_name` setting. |
+| matched_uri | URI of the Route that matches the request. Default to an empty string if a request does not match any Route. |
+| matched_host | Host of the Route that matches the request. Default to an empty string if a request does not match any Route, or host is not configured on the Route. |
+| service | ID of the Service that the HTTP status originates from when `prefer_name` is `false` (default), and name of the Service when `prefer_name` to `true`. Default to the configured value of host on the Route if the matched Route does not belong to any Service. |
+| service_id | Available only in Enterprise. ID of the Service that the HTTP status originates from regardless of the `prefer_name` setting. |
+| consumer | Name of the Consumer associated with a request. Default to an empty string if no Consumer is associated with the request. |
+| node | IP address of the upstream node. |
+| gateway_group_id | Available only in Enterprise. ID of the gateway group that the HTTP status originates from. |
+| instance_id | Available only in Enterprise. ID of the gateway instance that the HTTP status originates from. |
+| api_product_id | Available only in Enterprise. Product ID that the HTTP status originates from. |
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- expire: 86400
-```
+### Labels for `apisix_bandwidth`
-## Metrics endpoint
+The following labels are used to differentiate `apisix_bandwidth` metrics.
-This Plugin will add the metrics endpoint `/apisix/prometheus/metrics` or your custom export URI for exposing the metrics.
+| Name | Description |
+| ---------- | ----------------------------------------------------------------------------------------------------------------------------- |
+| type | Type of traffic, `egress` or `ingress`. |
+| route | ID of the Route that bandwidth corresponds to when `prefer_name` is `false` (default), and name of the Route when `prefer_name` to `true`. Default to an empty string if a request does not match any Route. |
+| route_id | Available only in Enterprise. ID of the Route that bandwidth corresponds to regardless of the `prefer_name` setting. |
+| service | ID of the Service that bandwidth corresponds to when `prefer_name` is `false` (default), and name of the Service when `prefer_name` to `true`. Default to the configured value of host on the Route if the matched Route does not belong to any Service. |
+| service_id | Available only in Enterprise. ID of the Service that bandwidth corresponds to regardless of the `prefer_name` setting. |
+| consumer | Name of the Consumer associated with a request. Default to an empty string if no Consumer is associated with the request. |
+| node | IP address of the upstream node. |
+| gateway_group_id | Available only in Enterprise. ID of the gateway group that bandwidth corresponds to. |
+| instance_id | Available only in Enterprise. ID of the gateway instance that bandwidth corresponds to. |
+| api_product_id | Available only in Enterprise. Product ID that bandwidth corresponds to. |
-These metrics are exposed by a separate Prometheus server address. By default, the address is `127.0.0.1:9091`. You can change it in your configuration file (`conf/config.yaml`):
+### Labels for `apisix_http_latency`
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- export_addr:
- ip: ${{INTRANET_IP}}
- port: 9092
-```
+The following labels are used to differentiate `apisix_http_latency` metrics.
-Now, if the environment variable `INTRANET_IP` is `172.1.1.1`, APISIX will export the metrics via `172.1.1.1:9092`.
+| Name | Description |
+| ---------- | ----------------------------------------------------------------------------------------------------------------------------------- |
+| type | Type of latencies. See [latency types](#latency-types) for details. |
+| route | ID of the Route that latencies correspond to when `prefer_name` is `false` (default), and name of the Route when `prefer_name` to `true`. Default to an empty string if a request does not match any Route. |
+| route_id | Available only in Enterprise. ID of the Route that latencies correspond to regardless of the `prefer_name` setting. |
+| service | ID of the Service that latencies correspond to when `prefer_name` is `false` (default), and name of the Service when `prefer_name` to `true`. Default to the configured value of host on the Route if the matched Route does not belong to any Service. |
+| service_id | Available only in Enterprise. ID of the Service that latencies correspond to regardless of the `prefer_name` setting. |
+| consumer | Name of the Consumer associated with latencies. Default to an empty string if no Consumer is associated with the request. |
+| node | IP address of the upstream node associated with latencies. |
+| gateway_group_id | Available only in Enterprise. ID of the gateway group that latencies correspond to. |
+| instance_id | Available only in Enterprise. ID of the gateway instance that latencies correspond to. |
+| api_product_id | Available only in Enterprise. Product ID that latencies correspond to. |
-If you still want to expose the metrics via the data plane port (default: `9080`), you can configure it as shown below:
+#### Latency Types
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- enable_export_server: false
+`apisix_http_latency` can be labeled with one of the three types:
+
+* `request` represents the time elapsed between the first byte was read from the client and the log write after the last byte was sent to the client.
+
+* `upstream` represents the time elapsed waiting on responses from the upstream Service.
+
+* `apisix` represents the difference between the `request` latency and `upstream` latency.
+
+In other words, the APISIX latency is not only attributed to the Lua processing. It should be understood as follows:
+
+```text
+APISIX latency
+ = downstream request time - upstream response time
+ = downstream traffic latency + NGINX latency
```
-You can then expose it by using the [public-api](public-api.md) Plugin.
+### Labels for `apisix_upstream_status`
-:::info IMPORTANT
+The following labels are used to differentiate `apisix_upstream_status` metrics.
-If the Prometheus plugin collects too many metrics, it will take CPU resources to calculate the metric data when getting the metrics via URI, which may affect APISIX to process normal requests. To solve this problem, APISIX exposes the URI and calculates the metrics in the [privileged agent](https://github.com/openresty/lua-resty-core/blob/master/lib/ngx/process.md#enable_privileged_agent).
-If the URI is exposed using the public-api plugin, then APISIX will calculate the metric data in a normal worker process, which may still affect APISIX processing of normal requests.
+| Name | Description |
+| ---------- | --------------------------------------------------------------------------------------------------- |
+| name | Resource ID corresponding to the upstream configured with health checks, such as `/apisix/routes/1` and `/apisix/upstreams/1`. |
+| ip | IP address of the upstream node. |
+| port | Port number of the node. |
-This feature requires APISIX to run on [APISIX-Runtime](../FAQ.md#how-do-i-build-the-apisix-runtime-environment).
+## Examples
-:::
+The examples below demonstrate how you can work with the `prometheus` Plugin for different scenarios.
-## Enable Plugin
+### Get APISIX Metrics
-The `prometheus` Plugin can be enabled with an empty table.
+The following example demonstrates how you can get metrics from APISIX.
-The example below shows how you can configure the Plugin on a specific Route:
+The default Prometheus metrics endpoint and other Prometheus related configurations can be found in the [static configuration](#static-configurations). If you would like to customize these configuration, see [configuration based on environments](https://apisix.apache.org/docs/apisix/profile/).
-:::note
-You can fetch the `admin_key` from `config.yaml` and save to an environment variable with the following command:
+If you deploy APISIX in a containerized environment and would like to access the Prometheus metrics endpoint externally, update the configuration file as follows and reload APISIX:
-```bash
-admin_key=$(yq '.deployment.admin.admin_key[0].key' conf/config.yaml | sed 's/"//g')
+```yaml title="conf/config.yaml"
+plugin_attr:
+ prometheus:
+ export_addr:
+ ip: 0.0.0.0
```
-:::
+Send a request to the APISIX Prometheus metrics endpoint:
```shell
-curl http://127.0.0.1:9180/apisix/admin/routes/1 -H "X-API-KEY: $admin_key" -X PUT -d '
-{
- "uri": "/hello",
- "plugins": {
- "prometheus":{}
- },
- "upstream": {
- "type": "roundrobin",
- "nodes": {
- "127.0.0.1:80": 1
- }
- }
-}'
+curl "http://127.0.0.1:9091/apisix/prometheus/metrics"
```
-:::note
-
-When `prefer_name` is set to `true` make sure to not duplicate names for multiple Routes/Services or it could be misleading.
-
-:::
+You should see an output similar to the following:
-<!-- You can use [APISIX Dashboard](https://github.com/apache/apisix-dashboard) to complete the above operations through the web console.
-
-First, add a Route:
+```text
+# HELP apisix_bandwidth Total bandwidth in bytes consumed per Service in Apisix
+# TYPE apisix_bandwidth counter
+apisix_bandwidth{type="egress",route="",service="",consumer="",node=""} 8417
+apisix_bandwidth{type="egress",route="1",service="",consumer="",node="127.0.0.1"} 1420
+apisix_bandwidth{type="egress",route="2",service="",consumer="",node="127.0.0.1"} 1420
+apisix_bandwidth{type="ingress",route="",service="",consumer="",node=""} 189
+apisix_bandwidth{type="ingress",route="1",service="",consumer="",node="127.0.0.1"} 332
+apisix_bandwidth{type="ingress",route="2",service="",consumer="",node="127.0.0.1"} 332
+# HELP apisix_etcd_modify_indexes Etcd modify index for APISIX keys
+# TYPE apisix_etcd_modify_indexes gauge
+apisix_etcd_modify_indexes{key="consumers"} 0
+apisix_etcd_modify_indexes{key="global_rules"} 0
+...
+```
-
+### Expose APISIX Metrics on Public API Endpoint
-Then add prometheus plugin:
+The following example demonstrates how you can disable the Prometheus export server that, by default, exposes an endpoint on port `9091`, and expose APISIX Prometheus metrics on a new public API endpoint on port `9080`, which APISIX uses to listen to other client requests.
- -->
+Disable the Prometheus export server in the configuration file and reload APISIX for changes to take effect:
-## Fetching metrics
+```yaml title="conf/config.yaml"
+plugin_attr:
+ prometheus:
+ enable_export_server: false
+```
-You can fetch the metrics from the specified export URI (default: `/apisix/prometheus/metrics`):
+Next, create a Route with [`public-api`](./public-api.md) Plugin and expose a public API endpoint for APISIX metrics:
```shell
-curl -i http://127.0.0.1:9091/apisix/prometheus/metrics
+curl "http://127.0.0.1:9180/apisix/admin/routes/prometheus-metrics" -X PUT \
+ -H "X-API-KEY: ${admin_key}" \
+ -d '{
+ "uri": "/apisix/prometheus/metrics",
+ "plugins": {
+ "public-api": {}
+ }
+ }'
```
-You can add this address to Prometheus to fetch the data:
+Send a request to the new metrics endpoint to verify:
-```yaml
-scrape_configs:
- - job_name: "apisix"
- scrape_interval: 15s # This value will be related to the time range of the rate function in Prometheus QL. The time range in the rate function should be at least twice this value.
- metrics_path: "/apisix/prometheus/metrics"
- static_configs:
- - targets: ["127.0.0.1:9091"]
+```shell
+curl "http://127.0.0.1:9080/apisix/prometheus/metrics"
```
-Now, you will be able to check the status in your Prometheus console:
-
-
+You should see an output similar to the following:
-
-
-## Using Grafana to graph the metrics
-
-Metrics exported by the `prometheus` Plugin can be graphed in Grafana using a drop in dashboard.
-
-To set it up, download [Grafana dashboard meta](https://github.com/apache/apisix/blob/master/docs/assets/other/json/apisix-grafana-dashboard.json) and import it in Grafana. Or, you can go to [Grafana official](https://grafana.com/grafana/dashboards/11719) for Grafana metadata.
+```text
+# HELP apisix_http_requests_total The total number of client requests since APISIX started
+# TYPE apisix_http_requests_total gauge
+apisix_http_requests_total 1
+# HELP apisix_nginx_http_current_connections Number of HTTP connections
+# TYPE apisix_nginx_http_current_connections gauge
+apisix_nginx_http_current_connections{state="accepted"} 1
+apisix_nginx_http_current_connections{state="active"} 1
+apisix_nginx_http_current_connections{state="handled"} 1
+apisix_nginx_http_current_connections{state="reading"} 0
+apisix_nginx_http_current_connections{state="waiting"} 0
+apisix_nginx_http_current_connections{state="writing"} 1
+...
+```
-
+### Integrate APISIX with Prometheus and Grafana
-
+To learn about how to collect APISIX metrics with Prometheus and visualize them in Grafana, see [how-to guide](/apisix/how-to-guide/observability/monitor-apisix-with-prometheus).
-
+### Monitor Upstream Health Statuses
-
+The following example demonstrates how to monitor the health status of upstream nodes.
-## Available HTTP metrics
+Create a Route with the `prometheus` Plugin and configure upstream active health checks:
-The following metrics are exported by the `prometheus` Plugin:
+```shell
+curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
+ -H "X-API-KEY: ${admin_key}" \
+ -d '{
+ "id": "prometheus-route",
+ "uri": "/get",
+ "plugins": {
+ "prometheus": {}
+ },
+ "upstream": {
+ "type": "roundrobin",
+ "nodes": {
+ "httpbin.org:80": 1,
+ "127.0.0.1:20001": 1
+ },
+ "checks": {
+ "active": {
+ "timeout": 5,
+ "http_path": "/status",
+ "healthy": {
+ "interval": 2,
+ "successes": 1
+ },
+ "unhealthy": {
+ "interval": 1,
+ "http_failures": 2
+ }
+ },
+ "passive": {
+ "healthy": {
+ "http_statuses": [200, 201],
+ "successes": 3
+ },
+ "unhealthy": {
+ "http_statuses": [500],
+ "http_failures": 3,
+ "tcp_failures": 3
+ }
+ }
+ }
+ }
+ }'
+```
-- Status code: HTTP status code returned from Upstream services. They are available for a single service and across all services.
+Send a request to the APISIX Prometheus metrics endpoint:
- The available attributes are:
+```shell
+curl "http://127.0.0.1:9091/apisix/prometheus/metrics"
+```
- | Name | Description |
- |--------------|-------------------------------------------------------------------------------------------------------------------------------|
- | code | HTTP status code returned by the upstream service. |
- | route | `route_id` of the matched Route with request. Defaults to an empty string if the Routes don't match. |
- | matched_uri | `uri` of the Route matching the request. Defaults to an empty string if the Routes don't match. |
- | matched_host | `host` of the Route matching the request. Defaults to an empty string if the Routes don't match. |
- | service | `service_id` of the Route matching the request. If the Route does not have a `service_id` configured, it defaults to `$host`. |
- | consumer | `consumer_name` of the Consumer matching the request. Defaults to an empty string if it does not match. |
- | node | IP address of the Upstream node. |
+You should see an output similar to the following:
-- Bandwidth: Total amount of traffic (ingress and egress) flowing through APISIX. Total bandwidth of a service can also be obtained.
+```text
+# HELP apisix_upstream_status upstream status from health check
+# TYPE apisix_upstream_status gauge
+apisix_upstream_status{name="/apisix/routes/1",ip="54.237.103.220",port="80"} 1
+apisix_upstream_status{name="/apisix/routes/1",ip="127.0.0.1",port="20001"} 0
+```
- The available attributes are:
+This shows that the upstream node `httpbin.org:80` is healthy and the upstream node `127.0.0.1:20001` is unhealthy.
- | Name | Description |
- |----------|-------------------------------------------------------------------------------------------------------------------------------|
- | type | Type of traffic (egress/ingress). |
- | route | `route_id` of the matched Route with request. Defaults to an empty string if the Routes don't match. |
- | service | `service_id` of the Route matching the request. If the Route does not have a `service_id` configured, it defaults to `$host`. |
- | consumer | `consumer_name` of the Consumer matching the request. Defaults to an empty string if it does not match. |
- | node | IP address of the Upstream node. |
+To learn more about how to configure active and passive health checks, see [health checks](/docs/apisix/tutorials/health-check/).
-- etcd reachability: A gauge type representing whether etcd can be reached by APISIX. A value of `1` represents reachable and `0` represents unreachable.
-- Connections: Nginx connection metrics like active, reading, writing, and number of accepted connections.
-- Batch process entries: A gauge type useful when Plugins like [syslog](./syslog.md), [http-logger](./http-logger.md), [tcp-logger](./tcp-logger.md), [udp-logger](./udp-logger.md), and [zipkin](./zipkin.md) use batch process to send data. Entries that hasn't been sent in batch process will be counted in the metrics.
-- Latency: Histogram of the request time per service in different dimensions.
+### Add Extra Labels for Metrics
- The available attributes are:
+The following example demonstrates how to add additional labels to metrics and use the [Nginx variable](https://nginx.org/en/docs/http/ngx_http_core_module.html) in label values.
- | Name | Description |
- |----------|-------------------------------------------------------------------------------------------------------------------------------------|
- | type | Value can be one of `apisix`, `upstream`, or `request`. This translates to latency caused by APISIX, Upstream, or both (their sum). |
- | route | `route_id` of the matched Route with request. Defaults to an empty string if the Routes don't match. |
- | service | `service_id` of the Route matching the request. If the Route does not have a `service_id` configured, it defaults to `$host`. |
- | consumer | `consumer_name` of the Consumer matching the request. Defaults to an empty string if it does not match. |
- | node | IP address of the Upstream node. |
+Currently, only the following metrics support extra labels:
-- Info: Information about the APISIX node.
-- Shared dict: The capacity and free space of all nginx.shared.DICT in APISIX.
+* apisix_http_status
+* apisix_http_latency
+* apisix_bandwidth
-- `apisix_upstream_status`: Health check result status of upstream nodes. A value of `1` represents healthy and `0` represents unhealthy.
+Include the following configurations in the [configuration based on environments](https://apisix.apache.org/docs/apisix/profile/) to add labels for metrics and reload APISIX for changes to take effect:
- The available attributes are:
+```yaml title="conf/config.yaml"
+plugin_attr:
+ prometheus: # Plugin: prometheus
+ metrics: # Create extra labels from the NGINX variables.
+ http_status:
+ extra_labels: # Set the extra labels for http_status metrics.
+ - upstream_addr: $upstream_addr # Add an extra upstream_addr label with value being the NGINX variable $upstream_addr.
+ - route_name: $route_name # Add an extra route_name label with value being the APISIX variable $route_name.
+```
- | Name | Description |
- |--------------|-------------------------------------------------------------------------------------------------------------------------------|
- | name | resource id where the upstream node is attached to, e.g. `/apisix/routes/1`, `/apisix/upstreams/1`. |
- | ip | ip address of the node. |
- | port | port number of the node. |
+Note that if you define a variable in the label value but it does not correspond to any existing [Nginx variable](https://nginx.org/en/docs/http/ngx_http_core_module.html), the label value will default to an empty string.
Review Comment:
Nginx variable and APISIX variable
Link APISIX variable to [this doc](https://apisix.apache.org/docs/apisix/apisix-variable/), using the relative link.
##########
docs/en/latest/plugins/prometheus.md:
##########
@@ -27,439 +27,471 @@ description: This document contains information about the Apache APISIX promethe
#
-->
-## Description
-
-The `prometheus` Plugin exports metrics in [Prometheus exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/#exposition-formats).
+<head>
+ <link rel="canonical" href="https://docs.api7.ai/hub/prometheus" />
+</head>
-## Attributes
+## Description
-| Name | Type | Required | Default | Description |
-| ----------- | ------- | -------- | ------- | --------------------------------------------------------------------------------- |
-| prefer_name | boolean | False | false | When set to `true`, prints Route/Service name instead of ID in Prometheus metric. |
+The `prometheus` Plugin provides the capability to integrate APISIX with [Prometheus](https://prometheus.io).
-### Specifying `export_uri`
+After enabling the Plugin, APISIX will start collecting relevant metrics, such as API requests and latencies, and exporting them in a [text-based exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/#exposition-formats) to Prometheus. You can then create event monitoring and alerting in Prometheus to monitor the health of your API gateway and APIs.
-You can change the default export URI by configuring the `export_uri` attribute under `plugin_attr` in your configuration file (`conf/config.yaml`).
+## Static Configurations
-| Name | Type | Default | Description |
-| ---------- | ------ | ---------------------------- | ------------------------------------- |
-| export_uri | string | "/apisix/prometheus/metrics" | URI to export the Prometheus metrics. |
+By default, `prometheus` configurations are pre-configured in the [default configuration](https://github.com/apache/apisix/blob/master/apisix/cli/config.lua).
-Here is a configuration example:
+To customize these values, add the corresponding configurations to `config.yaml`. For example:
-```yaml title="conf/config.yaml"
+```yaml
plugin_attr:
- prometheus:
- export_uri: /apisix/metrics
+ prometheus: # Plugin: prometheus attributes
+ export_uri: /apisix/prometheus/metrics # Set the URI for the Prometheus metrics endpoint.
+ metric_prefix: apisix_ # Set the prefix for Prometheus metrics generated by APISIX.
+ enable_export_server: true # Enable the Prometheus export server.
+ export_addr: # Set the address for the Prometheus export server.
+ ip: 127.0.0.1 # Set the IP.
+ port: 9091 # Set the port.
+ # metrics: # Create extra labels for metrics.
+ # http_status: # These metrics will be prefixed with `apisix_`.
+ # extra_labels: # Set the extra labels for http_status metrics.
+ # - upstream_addr: $upstream_addr
+ # - status: $upstream_status
+ # expire: 0 # The expiration time of metrics in seconds.
+ # 0 means the metrics will not expire.
+ # http_latency:
+ # extra_labels: # Set the extra labels for http_latency metrics.
+ # - upstream_addr: $upstream_addr
+ # expire: 0 # The expiration time of metrics in seconds.
+ # 0 means the metrics will not expire.
+ # bandwidth:
+ # extra_labels: # Set the extra labels for bandwidth metrics.
+ # - upstream_addr: $upstream_addr
+ # expire: 0 # The expiration time of metrics in seconds.
+ # 0 means the metrics will not expire.
+ # default_buckets: # Set the default buckets for the `http_latency` metrics histogram.
+ # - 10
+ # - 50
+ # - 100
+ # - 200
+ # - 500
+ # - 1000
+ # - 2000
+ # - 5000
+ # - 10000
+ # - 30000
+ # - 60000
+ # - 500
```
-### Specifying `metrics`
+You can use the [Nginx variable](https://nginx.org/en/docs/http/ngx_http_core_module.html) to create `extra_labels`. See [add extra labels](/hub/prometheus#add-extra-labels-for-metrics).
-For http request related metrics, you could specify extra labels, which match the APISIX variables.
+Reload APISIX for changes to take effect.
-If you specify label for nonexist APISIX variable, the label value would be "".
+## Attribute
-Currently, only below metrics are supported:
+| Name | Type | Required | Default | Valid values | Description |
+| ------------- | ------- | -------- | ------- | ------------ | ------------------------------------------ |
+| `prefer_name` | boolean | | False | | If true, export Route/Service name instead of their ID in Prometheus metrics. |
-* http_status
-* http_latency
-* bandwidth
+## Metrics
-Here is a configuration example:
+There are different types of metrics in Prometheus. To understand their differences, see [metrics types](https://prometheus.io/docs/concepts/metric_types/).
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- metrics:
- http_status:
- extra_labels:
- - upstream_addr: $upstream_addr
- - upstream_status: $upstream_status
-```
+The following metrics are exported by the `prometheus` Plugin by default. See [get APISIX metrics](#get-apisix-metrics) for an example. Note that some metrics, such as `apisix_batch_process_entries`, are not readily visible if there are no data.
-### Specifying `default_buckets`
+| Name | Type | Description |
+| ------------------------------ | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| apisix_bandwidth | counter | Total amount of traffic flowing through APISIX in bytes. |
+| apisix_etcd_modify_indexes | gauge | Number of changes to etcd by APISIX keys. |
+| apisix_batch_process_entries | gauge | Number of remaining entries in a batch when sending data in batches, such as with `http logger`, and other logging Plugins. |
+| apisix_etcd_reachable | gauge | Whether APISIX can reach etcd. A value of `1` represents reachable and `0` represents unreachable. |
+| apisix_http_status | counter | HTTP status codes returned from upstream Services. |
+| apisix_http_requests_total | gauge | Number of HTTP requests from clients. |
+| apisix_nginx_http_current_connections | gauge | Number of current connections with clients. |
+| apisix_nginx_metric_errors_total | counter | Total number of `nginx-lua-prometheus` errors. |
+| apisix_http_latency | histogram | HTTP request latency in milliseconds. |
+| apisix_node_info | gauge | Information of the APISIX node, such as host name. |
+| apisix_shared_dict_capacity_bytes | gauge | The total capacity of an [NGINX shared dictionary](https://github.com/openresty/lua-nginx-module#ngxshareddict). |
+| apisix_shared_dict_free_space_bytes | gauge | The remaining space in an [NGINX shared dictionary](https://github.com/openresty/lua-nginx-module#ngxshareddict). |
+| apisix_upstream_status | gauge | Health check status of upstream nodes, available if health checks are configured on the upstream. A value of `1` represents healthy and `0` represents unhealthy. |
+| apisix_stream_connection_total | counter | Total number of connections handled per stream Route. |
-`DEFAULT_BUCKETS` is the default value for bucket array in `http_latency` metrics.
+## Labels
-You can change the `DEFAULT_BUCKETS` by configuring `default_buckets` attribute in you configuration file.
+[Labels](https://prometheus.io/docs/practices/naming/#labels) are attributes of metrics that are used to differentiate metrics.
-Here is a configuration example:
+For example, the `apisix_http_status` metric can be labeled with `route` information to identify which Route the HTTP status originates from.
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- default_buckets:
- - 15
- - 55
- - 105
- - 205
- - 505
-```
+The following are labels for a non-exhaustive list of APISIX metrics and their descriptions.
-### Specifying `expire`
+### Labels for `apisix_http_status`
-`expire` sets the expiration time of `apisix_http_status`, `apisix_bandwidth`, and `apisix_http_latency` metrics in seconds. When set to 0, metrics will not expire.
+The following labels are used to differentiate `apisix_http_status` metrics.
-Here is a configuration example:
+| Name | Description |
+| ------------ | ----------------------------------------------------------------------------------------------------------------------------- |
+| code | HTTP response code returned by the upstream node. |
+| route | ID of the Route that the HTTP status originates from when `prefer_name` is `false` (default), and name of the Route when `prefer_name` to `true`. Default to an empty string if a request does not match any Route. |
+| route_id | Available only in Enterprise. ID of the Route that the HTTP status originates from regardless of the `prefer_name` setting. |
+| matched_uri | URI of the Route that matches the request. Default to an empty string if a request does not match any Route. |
+| matched_host | Host of the Route that matches the request. Default to an empty string if a request does not match any Route, or host is not configured on the Route. |
+| service | ID of the Service that the HTTP status originates from when `prefer_name` is `false` (default), and name of the Service when `prefer_name` to `true`. Default to the configured value of host on the Route if the matched Route does not belong to any Service. |
+| service_id | Available only in Enterprise. ID of the Service that the HTTP status originates from regardless of the `prefer_name` setting. |
+| consumer | Name of the Consumer associated with a request. Default to an empty string if no Consumer is associated with the request. |
+| node | IP address of the upstream node. |
+| gateway_group_id | Available only in Enterprise. ID of the gateway group that the HTTP status originates from. |
+| instance_id | Available only in Enterprise. ID of the gateway instance that the HTTP status originates from. |
+| api_product_id | Available only in Enterprise. Product ID that the HTTP status originates from. |
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- expire: 86400
-```
+### Labels for `apisix_bandwidth`
-## Metrics endpoint
+The following labels are used to differentiate `apisix_bandwidth` metrics.
-This Plugin will add the metrics endpoint `/apisix/prometheus/metrics` or your custom export URI for exposing the metrics.
+| Name | Description |
+| ---------- | ----------------------------------------------------------------------------------------------------------------------------- |
+| type | Type of traffic, `egress` or `ingress`. |
+| route | ID of the Route that bandwidth corresponds to when `prefer_name` is `false` (default), and name of the Route when `prefer_name` to `true`. Default to an empty string if a request does not match any Route. |
+| route_id | Available only in Enterprise. ID of the Route that bandwidth corresponds to regardless of the `prefer_name` setting. |
+| service | ID of the Service that bandwidth corresponds to when `prefer_name` is `false` (default), and name of the Service when `prefer_name` to `true`. Default to the configured value of host on the Route if the matched Route does not belong to any Service. |
+| service_id | Available only in Enterprise. ID of the Service that bandwidth corresponds to regardless of the `prefer_name` setting. |
+| consumer | Name of the Consumer associated with a request. Default to an empty string if no Consumer is associated with the request. |
+| node | IP address of the upstream node. |
+| gateway_group_id | Available only in Enterprise. ID of the gateway group that bandwidth corresponds to. |
+| instance_id | Available only in Enterprise. ID of the gateway instance that bandwidth corresponds to. |
+| api_product_id | Available only in Enterprise. Product ID that bandwidth corresponds to. |
-These metrics are exposed by a separate Prometheus server address. By default, the address is `127.0.0.1:9091`. You can change it in your configuration file (`conf/config.yaml`):
+### Labels for `apisix_http_latency`
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- export_addr:
- ip: ${{INTRANET_IP}}
- port: 9092
-```
+The following labels are used to differentiate `apisix_http_latency` metrics.
-Now, if the environment variable `INTRANET_IP` is `172.1.1.1`, APISIX will export the metrics via `172.1.1.1:9092`.
+| Name | Description |
+| ---------- | ----------------------------------------------------------------------------------------------------------------------------------- |
+| type | Type of latencies. See [latency types](#latency-types) for details. |
+| route | ID of the Route that latencies correspond to when `prefer_name` is `false` (default), and name of the Route when `prefer_name` to `true`. Default to an empty string if a request does not match any Route. |
+| route_id | Available only in Enterprise. ID of the Route that latencies correspond to regardless of the `prefer_name` setting. |
+| service | ID of the Service that latencies correspond to when `prefer_name` is `false` (default), and name of the Service when `prefer_name` to `true`. Default to the configured value of host on the Route if the matched Route does not belong to any Service. |
+| service_id | Available only in Enterprise. ID of the Service that latencies correspond to regardless of the `prefer_name` setting. |
+| consumer | Name of the Consumer associated with latencies. Default to an empty string if no Consumer is associated with the request. |
+| node | IP address of the upstream node associated with latencies. |
+| gateway_group_id | Available only in Enterprise. ID of the gateway group that latencies correspond to. |
+| instance_id | Available only in Enterprise. ID of the gateway instance that latencies correspond to. |
+| api_product_id | Available only in Enterprise. Product ID that latencies correspond to. |
-If you still want to expose the metrics via the data plane port (default: `9080`), you can configure it as shown below:
+#### Latency Types
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- enable_export_server: false
+`apisix_http_latency` can be labeled with one of the three types:
+
+* `request` represents the time elapsed between the first byte was read from the client and the log write after the last byte was sent to the client.
+
+* `upstream` represents the time elapsed waiting on responses from the upstream Service.
+
+* `apisix` represents the difference between the `request` latency and `upstream` latency.
+
+In other words, the APISIX latency is not only attributed to the Lua processing. It should be understood as follows:
+
+```text
+APISIX latency
+ = downstream request time - upstream response time
+ = downstream traffic latency + NGINX latency
```
-You can then expose it by using the [public-api](public-api.md) Plugin.
+### Labels for `apisix_upstream_status`
-:::info IMPORTANT
+The following labels are used to differentiate `apisix_upstream_status` metrics.
-If the Prometheus plugin collects too many metrics, it will take CPU resources to calculate the metric data when getting the metrics via URI, which may affect APISIX to process normal requests. To solve this problem, APISIX exposes the URI and calculates the metrics in the [privileged agent](https://github.com/openresty/lua-resty-core/blob/master/lib/ngx/process.md#enable_privileged_agent).
-If the URI is exposed using the public-api plugin, then APISIX will calculate the metric data in a normal worker process, which may still affect APISIX processing of normal requests.
+| Name | Description |
+| ---------- | --------------------------------------------------------------------------------------------------- |
+| name | Resource ID corresponding to the upstream configured with health checks, such as `/apisix/routes/1` and `/apisix/upstreams/1`. |
+| ip | IP address of the upstream node. |
+| port | Port number of the node. |
-This feature requires APISIX to run on [APISIX-Runtime](../FAQ.md#how-do-i-build-the-apisix-runtime-environment).
+## Examples
-:::
+The examples below demonstrate how you can work with the `prometheus` Plugin for different scenarios.
-## Enable Plugin
+### Get APISIX Metrics
-The `prometheus` Plugin can be enabled with an empty table.
+The following example demonstrates how you can get metrics from APISIX.
-The example below shows how you can configure the Plugin on a specific Route:
+The default Prometheus metrics endpoint and other Prometheus related configurations can be found in the [static configuration](#static-configurations). If you would like to customize these configuration, see [configuration based on environments](https://apisix.apache.org/docs/apisix/profile/).
-:::note
-You can fetch the `admin_key` from `config.yaml` and save to an environment variable with the following command:
+If you deploy APISIX in a containerized environment and would like to access the Prometheus metrics endpoint externally, update the configuration file as follows and reload APISIX:
-```bash
-admin_key=$(yq '.deployment.admin.admin_key[0].key' conf/config.yaml | sed 's/"//g')
+```yaml title="conf/config.yaml"
+plugin_attr:
+ prometheus:
+ export_addr:
+ ip: 0.0.0.0
```
-:::
+Send a request to the APISIX Prometheus metrics endpoint:
```shell
-curl http://127.0.0.1:9180/apisix/admin/routes/1 -H "X-API-KEY: $admin_key" -X PUT -d '
-{
- "uri": "/hello",
- "plugins": {
- "prometheus":{}
- },
- "upstream": {
- "type": "roundrobin",
- "nodes": {
- "127.0.0.1:80": 1
- }
- }
-}'
+curl "http://127.0.0.1:9091/apisix/prometheus/metrics"
```
-:::note
-
-When `prefer_name` is set to `true` make sure to not duplicate names for multiple Routes/Services or it could be misleading.
-
-:::
+You should see an output similar to the following:
-<!-- You can use [APISIX Dashboard](https://github.com/apache/apisix-dashboard) to complete the above operations through the web console.
-
-First, add a Route:
+```text
+# HELP apisix_bandwidth Total bandwidth in bytes consumed per Service in Apisix
+# TYPE apisix_bandwidth counter
+apisix_bandwidth{type="egress",route="",service="",consumer="",node=""} 8417
+apisix_bandwidth{type="egress",route="1",service="",consumer="",node="127.0.0.1"} 1420
+apisix_bandwidth{type="egress",route="2",service="",consumer="",node="127.0.0.1"} 1420
+apisix_bandwidth{type="ingress",route="",service="",consumer="",node=""} 189
+apisix_bandwidth{type="ingress",route="1",service="",consumer="",node="127.0.0.1"} 332
+apisix_bandwidth{type="ingress",route="2",service="",consumer="",node="127.0.0.1"} 332
+# HELP apisix_etcd_modify_indexes Etcd modify index for APISIX keys
+# TYPE apisix_etcd_modify_indexes gauge
+apisix_etcd_modify_indexes{key="consumers"} 0
+apisix_etcd_modify_indexes{key="global_rules"} 0
+...
+```
-
+### Expose APISIX Metrics on Public API Endpoint
-Then add prometheus plugin:
+The following example demonstrates how you can disable the Prometheus export server that, by default, exposes an endpoint on port `9091`, and expose APISIX Prometheus metrics on a new public API endpoint on port `9080`, which APISIX uses to listen to other client requests.
- -->
+Disable the Prometheus export server in the configuration file and reload APISIX for changes to take effect:
-## Fetching metrics
+```yaml title="conf/config.yaml"
+plugin_attr:
+ prometheus:
+ enable_export_server: false
+```
-You can fetch the metrics from the specified export URI (default: `/apisix/prometheus/metrics`):
+Next, create a Route with [`public-api`](./public-api.md) Plugin and expose a public API endpoint for APISIX metrics:
```shell
-curl -i http://127.0.0.1:9091/apisix/prometheus/metrics
+curl "http://127.0.0.1:9180/apisix/admin/routes/prometheus-metrics" -X PUT \
+ -H "X-API-KEY: ${admin_key}" \
+ -d '{
+ "uri": "/apisix/prometheus/metrics",
+ "plugins": {
+ "public-api": {}
+ }
+ }'
```
-You can add this address to Prometheus to fetch the data:
+Send a request to the new metrics endpoint to verify:
-```yaml
-scrape_configs:
- - job_name: "apisix"
- scrape_interval: 15s # This value will be related to the time range of the rate function in Prometheus QL. The time range in the rate function should be at least twice this value.
- metrics_path: "/apisix/prometheus/metrics"
- static_configs:
- - targets: ["127.0.0.1:9091"]
+```shell
+curl "http://127.0.0.1:9080/apisix/prometheus/metrics"
```
-Now, you will be able to check the status in your Prometheus console:
-
-
+You should see an output similar to the following:
-
-
-## Using Grafana to graph the metrics
-
-Metrics exported by the `prometheus` Plugin can be graphed in Grafana using a drop in dashboard.
-
-To set it up, download [Grafana dashboard meta](https://github.com/apache/apisix/blob/master/docs/assets/other/json/apisix-grafana-dashboard.json) and import it in Grafana. Or, you can go to [Grafana official](https://grafana.com/grafana/dashboards/11719) for Grafana metadata.
+```text
+# HELP apisix_http_requests_total The total number of client requests since APISIX started
+# TYPE apisix_http_requests_total gauge
+apisix_http_requests_total 1
+# HELP apisix_nginx_http_current_connections Number of HTTP connections
+# TYPE apisix_nginx_http_current_connections gauge
+apisix_nginx_http_current_connections{state="accepted"} 1
+apisix_nginx_http_current_connections{state="active"} 1
+apisix_nginx_http_current_connections{state="handled"} 1
+apisix_nginx_http_current_connections{state="reading"} 0
+apisix_nginx_http_current_connections{state="waiting"} 0
+apisix_nginx_http_current_connections{state="writing"} 1
+...
+```
-
+### Integrate APISIX with Prometheus and Grafana
-
+To learn about how to collect APISIX metrics with Prometheus and visualize them in Grafana, see [how-to guide](/apisix/how-to-guide/observability/monitor-apisix-with-prometheus).
-
+### Monitor Upstream Health Statuses
-
+The following example demonstrates how to monitor the health status of upstream nodes.
-## Available HTTP metrics
+Create a Route with the `prometheus` Plugin and configure upstream active health checks:
-The following metrics are exported by the `prometheus` Plugin:
+```shell
+curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
+ -H "X-API-KEY: ${admin_key}" \
+ -d '{
+ "id": "prometheus-route",
+ "uri": "/get",
+ "plugins": {
+ "prometheus": {}
+ },
+ "upstream": {
+ "type": "roundrobin",
+ "nodes": {
+ "httpbin.org:80": 1,
+ "127.0.0.1:20001": 1
+ },
+ "checks": {
+ "active": {
+ "timeout": 5,
+ "http_path": "/status",
+ "healthy": {
+ "interval": 2,
+ "successes": 1
+ },
+ "unhealthy": {
+ "interval": 1,
+ "http_failures": 2
+ }
+ },
+ "passive": {
+ "healthy": {
+ "http_statuses": [200, 201],
+ "successes": 3
+ },
+ "unhealthy": {
+ "http_statuses": [500],
+ "http_failures": 3,
+ "tcp_failures": 3
+ }
+ }
+ }
+ }
+ }'
+```
-- Status code: HTTP status code returned from Upstream services. They are available for a single service and across all services.
+Send a request to the APISIX Prometheus metrics endpoint:
- The available attributes are:
+```shell
+curl "http://127.0.0.1:9091/apisix/prometheus/metrics"
+```
- | Name | Description |
- |--------------|-------------------------------------------------------------------------------------------------------------------------------|
- | code | HTTP status code returned by the upstream service. |
- | route | `route_id` of the matched Route with request. Defaults to an empty string if the Routes don't match. |
- | matched_uri | `uri` of the Route matching the request. Defaults to an empty string if the Routes don't match. |
- | matched_host | `host` of the Route matching the request. Defaults to an empty string if the Routes don't match. |
- | service | `service_id` of the Route matching the request. If the Route does not have a `service_id` configured, it defaults to `$host`. |
- | consumer | `consumer_name` of the Consumer matching the request. Defaults to an empty string if it does not match. |
- | node | IP address of the Upstream node. |
+You should see an output similar to the following:
-- Bandwidth: Total amount of traffic (ingress and egress) flowing through APISIX. Total bandwidth of a service can also be obtained.
+```text
+# HELP apisix_upstream_status upstream status from health check
+# TYPE apisix_upstream_status gauge
+apisix_upstream_status{name="/apisix/routes/1",ip="54.237.103.220",port="80"} 1
+apisix_upstream_status{name="/apisix/routes/1",ip="127.0.0.1",port="20001"} 0
+```
- The available attributes are:
+This shows that the upstream node `httpbin.org:80` is healthy and the upstream node `127.0.0.1:20001` is unhealthy.
- | Name | Description |
- |----------|-------------------------------------------------------------------------------------------------------------------------------|
- | type | Type of traffic (egress/ingress). |
- | route | `route_id` of the matched Route with request. Defaults to an empty string if the Routes don't match. |
- | service | `service_id` of the Route matching the request. If the Route does not have a `service_id` configured, it defaults to `$host`. |
- | consumer | `consumer_name` of the Consumer matching the request. Defaults to an empty string if it does not match. |
- | node | IP address of the Upstream node. |
+To learn more about how to configure active and passive health checks, see [health checks](/docs/apisix/tutorials/health-check/).
Review Comment:
Broken link. Just remove this sentence.
##########
docs/en/latest/plugins/prometheus.md:
##########
@@ -27,439 +27,471 @@ description: This document contains information about the Apache APISIX promethe
#
-->
-## Description
-
-The `prometheus` Plugin exports metrics in [Prometheus exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/#exposition-formats).
+<head>
+ <link rel="canonical" href="https://docs.api7.ai/hub/prometheus" />
+</head>
-## Attributes
+## Description
-| Name | Type | Required | Default | Description |
-| ----------- | ------- | -------- | ------- | --------------------------------------------------------------------------------- |
-| prefer_name | boolean | False | false | When set to `true`, prints Route/Service name instead of ID in Prometheus metric. |
+The `prometheus` Plugin provides the capability to integrate APISIX with [Prometheus](https://prometheus.io).
-### Specifying `export_uri`
+After enabling the Plugin, APISIX will start collecting relevant metrics, such as API requests and latencies, and exporting them in a [text-based exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/#exposition-formats) to Prometheus. You can then create event monitoring and alerting in Prometheus to monitor the health of your API gateway and APIs.
-You can change the default export URI by configuring the `export_uri` attribute under `plugin_attr` in your configuration file (`conf/config.yaml`).
+## Static Configurations
-| Name | Type | Default | Description |
-| ---------- | ------ | ---------------------------- | ------------------------------------- |
-| export_uri | string | "/apisix/prometheus/metrics" | URI to export the Prometheus metrics. |
+By default, `prometheus` configurations are pre-configured in the [default configuration](https://github.com/apache/apisix/blob/master/apisix/cli/config.lua).
-Here is a configuration example:
+To customize these values, add the corresponding configurations to `config.yaml`. For example:
-```yaml title="conf/config.yaml"
+```yaml
plugin_attr:
- prometheus:
- export_uri: /apisix/metrics
+ prometheus: # Plugin: prometheus attributes
+ export_uri: /apisix/prometheus/metrics # Set the URI for the Prometheus metrics endpoint.
+ metric_prefix: apisix_ # Set the prefix for Prometheus metrics generated by APISIX.
+ enable_export_server: true # Enable the Prometheus export server.
+ export_addr: # Set the address for the Prometheus export server.
+ ip: 127.0.0.1 # Set the IP.
+ port: 9091 # Set the port.
+ # metrics: # Create extra labels for metrics.
+ # http_status: # These metrics will be prefixed with `apisix_`.
+ # extra_labels: # Set the extra labels for http_status metrics.
+ # - upstream_addr: $upstream_addr
+ # - status: $upstream_status
+ # expire: 0 # The expiration time of metrics in seconds.
+ # 0 means the metrics will not expire.
+ # http_latency:
+ # extra_labels: # Set the extra labels for http_latency metrics.
+ # - upstream_addr: $upstream_addr
+ # expire: 0 # The expiration time of metrics in seconds.
+ # 0 means the metrics will not expire.
+ # bandwidth:
+ # extra_labels: # Set the extra labels for bandwidth metrics.
+ # - upstream_addr: $upstream_addr
+ # expire: 0 # The expiration time of metrics in seconds.
+ # 0 means the metrics will not expire.
+ # default_buckets: # Set the default buckets for the `http_latency` metrics histogram.
+ # - 10
+ # - 50
+ # - 100
+ # - 200
+ # - 500
+ # - 1000
+ # - 2000
+ # - 5000
+ # - 10000
+ # - 30000
+ # - 60000
+ # - 500
```
-### Specifying `metrics`
+You can use the [Nginx variable](https://nginx.org/en/docs/http/ngx_http_core_module.html) to create `extra_labels`. See [add extra labels](/hub/prometheus#add-extra-labels-for-metrics).
-For http request related metrics, you could specify extra labels, which match the APISIX variables.
+Reload APISIX for changes to take effect.
-If you specify label for nonexist APISIX variable, the label value would be "".
+## Attribute
-Currently, only below metrics are supported:
+| Name | Type | Required | Default | Valid values | Description |
+| ------------- | ------- | -------- | ------- | ------------ | ------------------------------------------ |
+| `prefer_name` | boolean | | False | | If true, export Route/Service name instead of their ID in Prometheus metrics. |
-* http_status
-* http_latency
-* bandwidth
+## Metrics
-Here is a configuration example:
+There are different types of metrics in Prometheus. To understand their differences, see [metrics types](https://prometheus.io/docs/concepts/metric_types/).
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- metrics:
- http_status:
- extra_labels:
- - upstream_addr: $upstream_addr
- - upstream_status: $upstream_status
-```
+The following metrics are exported by the `prometheus` Plugin by default. See [get APISIX metrics](#get-apisix-metrics) for an example. Note that some metrics, such as `apisix_batch_process_entries`, are not readily visible if there are no data.
-### Specifying `default_buckets`
+| Name | Type | Description |
+| ------------------------------ | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| apisix_bandwidth | counter | Total amount of traffic flowing through APISIX in bytes. |
+| apisix_etcd_modify_indexes | gauge | Number of changes to etcd by APISIX keys. |
+| apisix_batch_process_entries | gauge | Number of remaining entries in a batch when sending data in batches, such as with `http logger`, and other logging Plugins. |
+| apisix_etcd_reachable | gauge | Whether APISIX can reach etcd. A value of `1` represents reachable and `0` represents unreachable. |
+| apisix_http_status | counter | HTTP status codes returned from upstream Services. |
+| apisix_http_requests_total | gauge | Number of HTTP requests from clients. |
+| apisix_nginx_http_current_connections | gauge | Number of current connections with clients. |
+| apisix_nginx_metric_errors_total | counter | Total number of `nginx-lua-prometheus` errors. |
+| apisix_http_latency | histogram | HTTP request latency in milliseconds. |
+| apisix_node_info | gauge | Information of the APISIX node, such as host name. |
+| apisix_shared_dict_capacity_bytes | gauge | The total capacity of an [NGINX shared dictionary](https://github.com/openresty/lua-nginx-module#ngxshareddict). |
+| apisix_shared_dict_free_space_bytes | gauge | The remaining space in an [NGINX shared dictionary](https://github.com/openresty/lua-nginx-module#ngxshareddict). |
+| apisix_upstream_status | gauge | Health check status of upstream nodes, available if health checks are configured on the upstream. A value of `1` represents healthy and `0` represents unhealthy. |
+| apisix_stream_connection_total | counter | Total number of connections handled per stream Route. |
-`DEFAULT_BUCKETS` is the default value for bucket array in `http_latency` metrics.
+## Labels
-You can change the `DEFAULT_BUCKETS` by configuring `default_buckets` attribute in you configuration file.
+[Labels](https://prometheus.io/docs/practices/naming/#labels) are attributes of metrics that are used to differentiate metrics.
-Here is a configuration example:
+For example, the `apisix_http_status` metric can be labeled with `route` information to identify which Route the HTTP status originates from.
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- default_buckets:
- - 15
- - 55
- - 105
- - 205
- - 505
-```
+The following are labels for a non-exhaustive list of APISIX metrics and their descriptions.
-### Specifying `expire`
+### Labels for `apisix_http_status`
-`expire` sets the expiration time of `apisix_http_status`, `apisix_bandwidth`, and `apisix_http_latency` metrics in seconds. When set to 0, metrics will not expire.
+The following labels are used to differentiate `apisix_http_status` metrics.
-Here is a configuration example:
+| Name | Description |
+| ------------ | ----------------------------------------------------------------------------------------------------------------------------- |
+| code | HTTP response code returned by the upstream node. |
+| route | ID of the Route that the HTTP status originates from when `prefer_name` is `false` (default), and name of the Route when `prefer_name` to `true`. Default to an empty string if a request does not match any Route. |
+| route_id | Available only in Enterprise. ID of the Route that the HTTP status originates from regardless of the `prefer_name` setting. |
+| matched_uri | URI of the Route that matches the request. Default to an empty string if a request does not match any Route. |
+| matched_host | Host of the Route that matches the request. Default to an empty string if a request does not match any Route, or host is not configured on the Route. |
+| service | ID of the Service that the HTTP status originates from when `prefer_name` is `false` (default), and name of the Service when `prefer_name` to `true`. Default to the configured value of host on the Route if the matched Route does not belong to any Service. |
+| service_id | Available only in Enterprise. ID of the Service that the HTTP status originates from regardless of the `prefer_name` setting. |
+| consumer | Name of the Consumer associated with a request. Default to an empty string if no Consumer is associated with the request. |
+| node | IP address of the upstream node. |
+| gateway_group_id | Available only in Enterprise. ID of the gateway group that the HTTP status originates from. |
+| instance_id | Available only in Enterprise. ID of the gateway instance that the HTTP status originates from. |
+| api_product_id | Available only in Enterprise. Product ID that the HTTP status originates from. |
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- expire: 86400
-```
+### Labels for `apisix_bandwidth`
-## Metrics endpoint
+The following labels are used to differentiate `apisix_bandwidth` metrics.
-This Plugin will add the metrics endpoint `/apisix/prometheus/metrics` or your custom export URI for exposing the metrics.
+| Name | Description |
+| ---------- | ----------------------------------------------------------------------------------------------------------------------------- |
+| type | Type of traffic, `egress` or `ingress`. |
+| route | ID of the Route that bandwidth corresponds to when `prefer_name` is `false` (default), and name of the Route when `prefer_name` to `true`. Default to an empty string if a request does not match any Route. |
+| route_id | Available only in Enterprise. ID of the Route that bandwidth corresponds to regardless of the `prefer_name` setting. |
+| service | ID of the Service that bandwidth corresponds to when `prefer_name` is `false` (default), and name of the Service when `prefer_name` to `true`. Default to the configured value of host on the Route if the matched Route does not belong to any Service. |
+| service_id | Available only in Enterprise. ID of the Service that bandwidth corresponds to regardless of the `prefer_name` setting. |
+| consumer | Name of the Consumer associated with a request. Default to an empty string if no Consumer is associated with the request. |
+| node | IP address of the upstream node. |
+| gateway_group_id | Available only in Enterprise. ID of the gateway group that bandwidth corresponds to. |
+| instance_id | Available only in Enterprise. ID of the gateway instance that bandwidth corresponds to. |
+| api_product_id | Available only in Enterprise. Product ID that bandwidth corresponds to. |
-These metrics are exposed by a separate Prometheus server address. By default, the address is `127.0.0.1:9091`. You can change it in your configuration file (`conf/config.yaml`):
+### Labels for `apisix_http_latency`
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- export_addr:
- ip: ${{INTRANET_IP}}
- port: 9092
-```
+The following labels are used to differentiate `apisix_http_latency` metrics.
-Now, if the environment variable `INTRANET_IP` is `172.1.1.1`, APISIX will export the metrics via `172.1.1.1:9092`.
+| Name | Description |
+| ---------- | ----------------------------------------------------------------------------------------------------------------------------------- |
+| type | Type of latencies. See [latency types](#latency-types) for details. |
+| route | ID of the Route that latencies correspond to when `prefer_name` is `false` (default), and name of the Route when `prefer_name` to `true`. Default to an empty string if a request does not match any Route. |
+| route_id | Available only in Enterprise. ID of the Route that latencies correspond to regardless of the `prefer_name` setting. |
+| service | ID of the Service that latencies correspond to when `prefer_name` is `false` (default), and name of the Service when `prefer_name` to `true`. Default to the configured value of host on the Route if the matched Route does not belong to any Service. |
+| service_id | Available only in Enterprise. ID of the Service that latencies correspond to regardless of the `prefer_name` setting. |
+| consumer | Name of the Consumer associated with latencies. Default to an empty string if no Consumer is associated with the request. |
+| node | IP address of the upstream node associated with latencies. |
+| gateway_group_id | Available only in Enterprise. ID of the gateway group that latencies correspond to. |
+| instance_id | Available only in Enterprise. ID of the gateway instance that latencies correspond to. |
+| api_product_id | Available only in Enterprise. Product ID that latencies correspond to. |
-If you still want to expose the metrics via the data plane port (default: `9080`), you can configure it as shown below:
+#### Latency Types
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- enable_export_server: false
+`apisix_http_latency` can be labeled with one of the three types:
+
+* `request` represents the time elapsed between the first byte was read from the client and the log write after the last byte was sent to the client.
+
+* `upstream` represents the time elapsed waiting on responses from the upstream Service.
+
+* `apisix` represents the difference between the `request` latency and `upstream` latency.
+
+In other words, the APISIX latency is not only attributed to the Lua processing. It should be understood as follows:
+
+```text
+APISIX latency
+ = downstream request time - upstream response time
+ = downstream traffic latency + NGINX latency
```
-You can then expose it by using the [public-api](public-api.md) Plugin.
+### Labels for `apisix_upstream_status`
-:::info IMPORTANT
+The following labels are used to differentiate `apisix_upstream_status` metrics.
-If the Prometheus plugin collects too many metrics, it will take CPU resources to calculate the metric data when getting the metrics via URI, which may affect APISIX to process normal requests. To solve this problem, APISIX exposes the URI and calculates the metrics in the [privileged agent](https://github.com/openresty/lua-resty-core/blob/master/lib/ngx/process.md#enable_privileged_agent).
-If the URI is exposed using the public-api plugin, then APISIX will calculate the metric data in a normal worker process, which may still affect APISIX processing of normal requests.
+| Name | Description |
+| ---------- | --------------------------------------------------------------------------------------------------- |
+| name | Resource ID corresponding to the upstream configured with health checks, such as `/apisix/routes/1` and `/apisix/upstreams/1`. |
+| ip | IP address of the upstream node. |
+| port | Port number of the node. |
-This feature requires APISIX to run on [APISIX-Runtime](../FAQ.md#how-do-i-build-the-apisix-runtime-environment).
+## Examples
-:::
+The examples below demonstrate how you can work with the `prometheus` Plugin for different scenarios.
-## Enable Plugin
+### Get APISIX Metrics
-The `prometheus` Plugin can be enabled with an empty table.
+The following example demonstrates how you can get metrics from APISIX.
-The example below shows how you can configure the Plugin on a specific Route:
+The default Prometheus metrics endpoint and other Prometheus related configurations can be found in the [static configuration](#static-configurations). If you would like to customize these configuration, see [configuration based on environments](https://apisix.apache.org/docs/apisix/profile/).
Review Comment:
The `configuration based on environments` doc is irrelevant.
```suggestion
The default Prometheus metrics endpoint and other Prometheus related configurations can be found in the [static configuration](#static-configurations). If you would like to customize these configurations, update `config.yaml` and reload APISIX.
```
##########
docs/en/latest/plugins/prometheus.md:
##########
@@ -27,439 +27,471 @@ description: This document contains information about the Apache APISIX promethe
#
-->
-## Description
-
-The `prometheus` Plugin exports metrics in [Prometheus exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/#exposition-formats).
+<head>
+ <link rel="canonical" href="https://docs.api7.ai/hub/prometheus" />
+</head>
-## Attributes
+## Description
-| Name | Type | Required | Default | Description |
-| ----------- | ------- | -------- | ------- | --------------------------------------------------------------------------------- |
-| prefer_name | boolean | False | false | When set to `true`, prints Route/Service name instead of ID in Prometheus metric. |
+The `prometheus` Plugin provides the capability to integrate APISIX with [Prometheus](https://prometheus.io).
-### Specifying `export_uri`
+After enabling the Plugin, APISIX will start collecting relevant metrics, such as API requests and latencies, and exporting them in a [text-based exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/#exposition-formats) to Prometheus. You can then create event monitoring and alerting in Prometheus to monitor the health of your API gateway and APIs.
-You can change the default export URI by configuring the `export_uri` attribute under `plugin_attr` in your configuration file (`conf/config.yaml`).
+## Static Configurations
-| Name | Type | Default | Description |
-| ---------- | ------ | ---------------------------- | ------------------------------------- |
-| export_uri | string | "/apisix/prometheus/metrics" | URI to export the Prometheus metrics. |
+By default, `prometheus` configurations are pre-configured in the [default configuration](https://github.com/apache/apisix/blob/master/apisix/cli/config.lua).
-Here is a configuration example:
+To customize these values, add the corresponding configurations to `config.yaml`. For example:
-```yaml title="conf/config.yaml"
+```yaml
plugin_attr:
- prometheus:
- export_uri: /apisix/metrics
+ prometheus: # Plugin: prometheus attributes
+ export_uri: /apisix/prometheus/metrics # Set the URI for the Prometheus metrics endpoint.
+ metric_prefix: apisix_ # Set the prefix for Prometheus metrics generated by APISIX.
+ enable_export_server: true # Enable the Prometheus export server.
+ export_addr: # Set the address for the Prometheus export server.
+ ip: 127.0.0.1 # Set the IP.
+ port: 9091 # Set the port.
+ # metrics: # Create extra labels for metrics.
+ # http_status: # These metrics will be prefixed with `apisix_`.
+ # extra_labels: # Set the extra labels for http_status metrics.
+ # - upstream_addr: $upstream_addr
+ # - status: $upstream_status
+ # expire: 0 # The expiration time of metrics in seconds.
+ # 0 means the metrics will not expire.
+ # http_latency:
+ # extra_labels: # Set the extra labels for http_latency metrics.
+ # - upstream_addr: $upstream_addr
+ # expire: 0 # The expiration time of metrics in seconds.
+ # 0 means the metrics will not expire.
+ # bandwidth:
+ # extra_labels: # Set the extra labels for bandwidth metrics.
+ # - upstream_addr: $upstream_addr
+ # expire: 0 # The expiration time of metrics in seconds.
+ # 0 means the metrics will not expire.
+ # default_buckets: # Set the default buckets for the `http_latency` metrics histogram.
+ # - 10
+ # - 50
+ # - 100
+ # - 200
+ # - 500
+ # - 1000
+ # - 2000
+ # - 5000
+ # - 10000
+ # - 30000
+ # - 60000
+ # - 500
```
-### Specifying `metrics`
+You can use the [Nginx variable](https://nginx.org/en/docs/http/ngx_http_core_module.html) to create `extra_labels`. See [add extra labels](/hub/prometheus#add-extra-labels-for-metrics).
-For http request related metrics, you could specify extra labels, which match the APISIX variables.
+Reload APISIX for changes to take effect.
-If you specify label for nonexist APISIX variable, the label value would be "".
+## Attribute
-Currently, only below metrics are supported:
+| Name | Type | Required | Default | Valid values | Description |
+| ------------- | ------- | -------- | ------- | ------------ | ------------------------------------------ |
+| `prefer_name` | boolean | | False | | If true, export Route/Service name instead of their ID in Prometheus metrics. |
-* http_status
-* http_latency
-* bandwidth
+## Metrics
-Here is a configuration example:
+There are different types of metrics in Prometheus. To understand their differences, see [metrics types](https://prometheus.io/docs/concepts/metric_types/).
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- metrics:
- http_status:
- extra_labels:
- - upstream_addr: $upstream_addr
- - upstream_status: $upstream_status
-```
+The following metrics are exported by the `prometheus` Plugin by default. See [get APISIX metrics](#get-apisix-metrics) for an example. Note that some metrics, such as `apisix_batch_process_entries`, are not readily visible if there are no data.
-### Specifying `default_buckets`
+| Name | Type | Description |
+| ------------------------------ | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| apisix_bandwidth | counter | Total amount of traffic flowing through APISIX in bytes. |
+| apisix_etcd_modify_indexes | gauge | Number of changes to etcd by APISIX keys. |
+| apisix_batch_process_entries | gauge | Number of remaining entries in a batch when sending data in batches, such as with `http logger`, and other logging Plugins. |
+| apisix_etcd_reachable | gauge | Whether APISIX can reach etcd. A value of `1` represents reachable and `0` represents unreachable. |
+| apisix_http_status | counter | HTTP status codes returned from upstream Services. |
+| apisix_http_requests_total | gauge | Number of HTTP requests from clients. |
+| apisix_nginx_http_current_connections | gauge | Number of current connections with clients. |
+| apisix_nginx_metric_errors_total | counter | Total number of `nginx-lua-prometheus` errors. |
+| apisix_http_latency | histogram | HTTP request latency in milliseconds. |
+| apisix_node_info | gauge | Information of the APISIX node, such as host name. |
+| apisix_shared_dict_capacity_bytes | gauge | The total capacity of an [NGINX shared dictionary](https://github.com/openresty/lua-nginx-module#ngxshareddict). |
+| apisix_shared_dict_free_space_bytes | gauge | The remaining space in an [NGINX shared dictionary](https://github.com/openresty/lua-nginx-module#ngxshareddict). |
+| apisix_upstream_status | gauge | Health check status of upstream nodes, available if health checks are configured on the upstream. A value of `1` represents healthy and `0` represents unhealthy. |
+| apisix_stream_connection_total | counter | Total number of connections handled per stream Route. |
-`DEFAULT_BUCKETS` is the default value for bucket array in `http_latency` metrics.
+## Labels
-You can change the `DEFAULT_BUCKETS` by configuring `default_buckets` attribute in you configuration file.
+[Labels](https://prometheus.io/docs/practices/naming/#labels) are attributes of metrics that are used to differentiate metrics.
-Here is a configuration example:
+For example, the `apisix_http_status` metric can be labeled with `route` information to identify which Route the HTTP status originates from.
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- default_buckets:
- - 15
- - 55
- - 105
- - 205
- - 505
-```
+The following are labels for a non-exhaustive list of APISIX metrics and their descriptions.
-### Specifying `expire`
+### Labels for `apisix_http_status`
-`expire` sets the expiration time of `apisix_http_status`, `apisix_bandwidth`, and `apisix_http_latency` metrics in seconds. When set to 0, metrics will not expire.
+The following labels are used to differentiate `apisix_http_status` metrics.
-Here is a configuration example:
+| Name | Description |
+| ------------ | ----------------------------------------------------------------------------------------------------------------------------- |
+| code | HTTP response code returned by the upstream node. |
+| route | ID of the Route that the HTTP status originates from when `prefer_name` is `false` (default), and name of the Route when `prefer_name` to `true`. Default to an empty string if a request does not match any Route. |
+| route_id | Available only in Enterprise. ID of the Route that the HTTP status originates from regardless of the `prefer_name` setting. |
+| matched_uri | URI of the Route that matches the request. Default to an empty string if a request does not match any Route. |
+| matched_host | Host of the Route that matches the request. Default to an empty string if a request does not match any Route, or host is not configured on the Route. |
+| service | ID of the Service that the HTTP status originates from when `prefer_name` is `false` (default), and name of the Service when `prefer_name` to `true`. Default to the configured value of host on the Route if the matched Route does not belong to any Service. |
+| service_id | Available only in Enterprise. ID of the Service that the HTTP status originates from regardless of the `prefer_name` setting. |
+| consumer | Name of the Consumer associated with a request. Default to an empty string if no Consumer is associated with the request. |
+| node | IP address of the upstream node. |
+| gateway_group_id | Available only in Enterprise. ID of the gateway group that the HTTP status originates from. |
+| instance_id | Available only in Enterprise. ID of the gateway instance that the HTTP status originates from. |
+| api_product_id | Available only in Enterprise. Product ID that the HTTP status originates from. |
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- expire: 86400
-```
+### Labels for `apisix_bandwidth`
-## Metrics endpoint
+The following labels are used to differentiate `apisix_bandwidth` metrics.
-This Plugin will add the metrics endpoint `/apisix/prometheus/metrics` or your custom export URI for exposing the metrics.
+| Name | Description |
+| ---------- | ----------------------------------------------------------------------------------------------------------------------------- |
+| type | Type of traffic, `egress` or `ingress`. |
+| route | ID of the Route that bandwidth corresponds to when `prefer_name` is `false` (default), and name of the Route when `prefer_name` to `true`. Default to an empty string if a request does not match any Route. |
+| route_id | Available only in Enterprise. ID of the Route that bandwidth corresponds to regardless of the `prefer_name` setting. |
+| service | ID of the Service that bandwidth corresponds to when `prefer_name` is `false` (default), and name of the Service when `prefer_name` to `true`. Default to the configured value of host on the Route if the matched Route does not belong to any Service. |
+| service_id | Available only in Enterprise. ID of the Service that bandwidth corresponds to regardless of the `prefer_name` setting. |
+| consumer | Name of the Consumer associated with a request. Default to an empty string if no Consumer is associated with the request. |
+| node | IP address of the upstream node. |
+| gateway_group_id | Available only in Enterprise. ID of the gateway group that bandwidth corresponds to. |
+| instance_id | Available only in Enterprise. ID of the gateway instance that bandwidth corresponds to. |
+| api_product_id | Available only in Enterprise. Product ID that bandwidth corresponds to. |
-These metrics are exposed by a separate Prometheus server address. By default, the address is `127.0.0.1:9091`. You can change it in your configuration file (`conf/config.yaml`):
+### Labels for `apisix_http_latency`
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- export_addr:
- ip: ${{INTRANET_IP}}
- port: 9092
-```
+The following labels are used to differentiate `apisix_http_latency` metrics.
-Now, if the environment variable `INTRANET_IP` is `172.1.1.1`, APISIX will export the metrics via `172.1.1.1:9092`.
+| Name | Description |
+| ---------- | ----------------------------------------------------------------------------------------------------------------------------------- |
+| type | Type of latencies. See [latency types](#latency-types) for details. |
+| route | ID of the Route that latencies correspond to when `prefer_name` is `false` (default), and name of the Route when `prefer_name` to `true`. Default to an empty string if a request does not match any Route. |
+| route_id | Available only in Enterprise. ID of the Route that latencies correspond to regardless of the `prefer_name` setting. |
+| service | ID of the Service that latencies correspond to when `prefer_name` is `false` (default), and name of the Service when `prefer_name` to `true`. Default to the configured value of host on the Route if the matched Route does not belong to any Service. |
+| service_id | Available only in Enterprise. ID of the Service that latencies correspond to regardless of the `prefer_name` setting. |
+| consumer | Name of the Consumer associated with latencies. Default to an empty string if no Consumer is associated with the request. |
+| node | IP address of the upstream node associated with latencies. |
+| gateway_group_id | Available only in Enterprise. ID of the gateway group that latencies correspond to. |
+| instance_id | Available only in Enterprise. ID of the gateway instance that latencies correspond to. |
+| api_product_id | Available only in Enterprise. Product ID that latencies correspond to. |
-If you still want to expose the metrics via the data plane port (default: `9080`), you can configure it as shown below:
+#### Latency Types
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- enable_export_server: false
+`apisix_http_latency` can be labeled with one of the three types:
+
+* `request` represents the time elapsed between the first byte was read from the client and the log write after the last byte was sent to the client.
+
+* `upstream` represents the time elapsed waiting on responses from the upstream Service.
+
+* `apisix` represents the difference between the `request` latency and `upstream` latency.
+
+In other words, the APISIX latency is not only attributed to the Lua processing. It should be understood as follows:
+
+```text
+APISIX latency
+ = downstream request time - upstream response time
+ = downstream traffic latency + NGINX latency
```
-You can then expose it by using the [public-api](public-api.md) Plugin.
+### Labels for `apisix_upstream_status`
-:::info IMPORTANT
+The following labels are used to differentiate `apisix_upstream_status` metrics.
-If the Prometheus plugin collects too many metrics, it will take CPU resources to calculate the metric data when getting the metrics via URI, which may affect APISIX to process normal requests. To solve this problem, APISIX exposes the URI and calculates the metrics in the [privileged agent](https://github.com/openresty/lua-resty-core/blob/master/lib/ngx/process.md#enable_privileged_agent).
-If the URI is exposed using the public-api plugin, then APISIX will calculate the metric data in a normal worker process, which may still affect APISIX processing of normal requests.
+| Name | Description |
+| ---------- | --------------------------------------------------------------------------------------------------- |
+| name | Resource ID corresponding to the upstream configured with health checks, such as `/apisix/routes/1` and `/apisix/upstreams/1`. |
+| ip | IP address of the upstream node. |
+| port | Port number of the node. |
-This feature requires APISIX to run on [APISIX-Runtime](../FAQ.md#how-do-i-build-the-apisix-runtime-environment).
+## Examples
-:::
+The examples below demonstrate how you can work with the `prometheus` Plugin for different scenarios.
-## Enable Plugin
+### Get APISIX Metrics
-The `prometheus` Plugin can be enabled with an empty table.
+The following example demonstrates how you can get metrics from APISIX.
-The example below shows how you can configure the Plugin on a specific Route:
+The default Prometheus metrics endpoint and other Prometheus related configurations can be found in the [static configuration](#static-configurations). If you would like to customize these configuration, see [configuration based on environments](https://apisix.apache.org/docs/apisix/profile/).
-:::note
-You can fetch the `admin_key` from `config.yaml` and save to an environment variable with the following command:
+If you deploy APISIX in a containerized environment and would like to access the Prometheus metrics endpoint externally, update the configuration file as follows and reload APISIX:
-```bash
-admin_key=$(yq '.deployment.admin.admin_key[0].key' conf/config.yaml | sed 's/"//g')
+```yaml title="conf/config.yaml"
+plugin_attr:
+ prometheus:
+ export_addr:
+ ip: 0.0.0.0
```
-:::
+Send a request to the APISIX Prometheus metrics endpoint:
```shell
-curl http://127.0.0.1:9180/apisix/admin/routes/1 -H "X-API-KEY: $admin_key" -X PUT -d '
-{
- "uri": "/hello",
- "plugins": {
- "prometheus":{}
- },
- "upstream": {
- "type": "roundrobin",
- "nodes": {
- "127.0.0.1:80": 1
- }
- }
-}'
+curl "http://127.0.0.1:9091/apisix/prometheus/metrics"
```
-:::note
-
-When `prefer_name` is set to `true` make sure to not duplicate names for multiple Routes/Services or it could be misleading.
-
-:::
+You should see an output similar to the following:
-<!-- You can use [APISIX Dashboard](https://github.com/apache/apisix-dashboard) to complete the above operations through the web console.
-
-First, add a Route:
+```text
+# HELP apisix_bandwidth Total bandwidth in bytes consumed per Service in Apisix
+# TYPE apisix_bandwidth counter
+apisix_bandwidth{type="egress",route="",service="",consumer="",node=""} 8417
+apisix_bandwidth{type="egress",route="1",service="",consumer="",node="127.0.0.1"} 1420
+apisix_bandwidth{type="egress",route="2",service="",consumer="",node="127.0.0.1"} 1420
+apisix_bandwidth{type="ingress",route="",service="",consumer="",node=""} 189
+apisix_bandwidth{type="ingress",route="1",service="",consumer="",node="127.0.0.1"} 332
+apisix_bandwidth{type="ingress",route="2",service="",consumer="",node="127.0.0.1"} 332
+# HELP apisix_etcd_modify_indexes Etcd modify index for APISIX keys
+# TYPE apisix_etcd_modify_indexes gauge
+apisix_etcd_modify_indexes{key="consumers"} 0
+apisix_etcd_modify_indexes{key="global_rules"} 0
+...
+```
-
+### Expose APISIX Metrics on Public API Endpoint
-Then add prometheus plugin:
+The following example demonstrates how you can disable the Prometheus export server that, by default, exposes an endpoint on port `9091`, and expose APISIX Prometheus metrics on a new public API endpoint on port `9080`, which APISIX uses to listen to other client requests.
- -->
+Disable the Prometheus export server in the configuration file and reload APISIX for changes to take effect:
-## Fetching metrics
+```yaml title="conf/config.yaml"
+plugin_attr:
+ prometheus:
+ enable_export_server: false
+```
-You can fetch the metrics from the specified export URI (default: `/apisix/prometheus/metrics`):
+Next, create a Route with [`public-api`](./public-api.md) Plugin and expose a public API endpoint for APISIX metrics:
```shell
-curl -i http://127.0.0.1:9091/apisix/prometheus/metrics
+curl "http://127.0.0.1:9180/apisix/admin/routes/prometheus-metrics" -X PUT \
+ -H "X-API-KEY: ${admin_key}" \
+ -d '{
+ "uri": "/apisix/prometheus/metrics",
+ "plugins": {
+ "public-api": {}
+ }
+ }'
```
-You can add this address to Prometheus to fetch the data:
+Send a request to the new metrics endpoint to verify:
-```yaml
-scrape_configs:
- - job_name: "apisix"
- scrape_interval: 15s # This value will be related to the time range of the rate function in Prometheus QL. The time range in the rate function should be at least twice this value.
- metrics_path: "/apisix/prometheus/metrics"
- static_configs:
- - targets: ["127.0.0.1:9091"]
+```shell
+curl "http://127.0.0.1:9080/apisix/prometheus/metrics"
```
-Now, you will be able to check the status in your Prometheus console:
-
-
+You should see an output similar to the following:
-
-
-## Using Grafana to graph the metrics
-
-Metrics exported by the `prometheus` Plugin can be graphed in Grafana using a drop in dashboard.
-
-To set it up, download [Grafana dashboard meta](https://github.com/apache/apisix/blob/master/docs/assets/other/json/apisix-grafana-dashboard.json) and import it in Grafana. Or, you can go to [Grafana official](https://grafana.com/grafana/dashboards/11719) for Grafana metadata.
+```text
+# HELP apisix_http_requests_total The total number of client requests since APISIX started
+# TYPE apisix_http_requests_total gauge
+apisix_http_requests_total 1
+# HELP apisix_nginx_http_current_connections Number of HTTP connections
+# TYPE apisix_nginx_http_current_connections gauge
+apisix_nginx_http_current_connections{state="accepted"} 1
+apisix_nginx_http_current_connections{state="active"} 1
+apisix_nginx_http_current_connections{state="handled"} 1
+apisix_nginx_http_current_connections{state="reading"} 0
+apisix_nginx_http_current_connections{state="waiting"} 0
+apisix_nginx_http_current_connections{state="writing"} 1
+...
+```
-
+### Integrate APISIX with Prometheus and Grafana
-
+To learn about how to collect APISIX metrics with Prometheus and visualize them in Grafana, see [how-to guide](/apisix/how-to-guide/observability/monitor-apisix-with-prometheus).
-
+### Monitor Upstream Health Statuses
-
+The following example demonstrates how to monitor the health status of upstream nodes.
-## Available HTTP metrics
+Create a Route with the `prometheus` Plugin and configure upstream active health checks:
-The following metrics are exported by the `prometheus` Plugin:
+```shell
+curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
+ -H "X-API-KEY: ${admin_key}" \
+ -d '{
+ "id": "prometheus-route",
+ "uri": "/get",
+ "plugins": {
+ "prometheus": {}
+ },
+ "upstream": {
+ "type": "roundrobin",
+ "nodes": {
+ "httpbin.org:80": 1,
+ "127.0.0.1:20001": 1
+ },
+ "checks": {
+ "active": {
+ "timeout": 5,
+ "http_path": "/status",
+ "healthy": {
+ "interval": 2,
+ "successes": 1
+ },
+ "unhealthy": {
+ "interval": 1,
+ "http_failures": 2
+ }
+ },
+ "passive": {
+ "healthy": {
+ "http_statuses": [200, 201],
+ "successes": 3
+ },
+ "unhealthy": {
+ "http_statuses": [500],
+ "http_failures": 3,
+ "tcp_failures": 3
+ }
+ }
+ }
+ }
+ }'
+```
-- Status code: HTTP status code returned from Upstream services. They are available for a single service and across all services.
+Send a request to the APISIX Prometheus metrics endpoint:
- The available attributes are:
+```shell
+curl "http://127.0.0.1:9091/apisix/prometheus/metrics"
+```
- | Name | Description |
- |--------------|-------------------------------------------------------------------------------------------------------------------------------|
- | code | HTTP status code returned by the upstream service. |
- | route | `route_id` of the matched Route with request. Defaults to an empty string if the Routes don't match. |
- | matched_uri | `uri` of the Route matching the request. Defaults to an empty string if the Routes don't match. |
- | matched_host | `host` of the Route matching the request. Defaults to an empty string if the Routes don't match. |
- | service | `service_id` of the Route matching the request. If the Route does not have a `service_id` configured, it defaults to `$host`. |
- | consumer | `consumer_name` of the Consumer matching the request. Defaults to an empty string if it does not match. |
- | node | IP address of the Upstream node. |
+You should see an output similar to the following:
-- Bandwidth: Total amount of traffic (ingress and egress) flowing through APISIX. Total bandwidth of a service can also be obtained.
+```text
+# HELP apisix_upstream_status upstream status from health check
+# TYPE apisix_upstream_status gauge
+apisix_upstream_status{name="/apisix/routes/1",ip="54.237.103.220",port="80"} 1
+apisix_upstream_status{name="/apisix/routes/1",ip="127.0.0.1",port="20001"} 0
+```
- The available attributes are:
+This shows that the upstream node `httpbin.org:80` is healthy and the upstream node `127.0.0.1:20001` is unhealthy.
- | Name | Description |
- |----------|-------------------------------------------------------------------------------------------------------------------------------|
- | type | Type of traffic (egress/ingress). |
- | route | `route_id` of the matched Route with request. Defaults to an empty string if the Routes don't match. |
- | service | `service_id` of the Route matching the request. If the Route does not have a `service_id` configured, it defaults to `$host`. |
- | consumer | `consumer_name` of the Consumer matching the request. Defaults to an empty string if it does not match. |
- | node | IP address of the Upstream node. |
+To learn more about how to configure active and passive health checks, see [health checks](/docs/apisix/tutorials/health-check/).
-- etcd reachability: A gauge type representing whether etcd can be reached by APISIX. A value of `1` represents reachable and `0` represents unreachable.
-- Connections: Nginx connection metrics like active, reading, writing, and number of accepted connections.
-- Batch process entries: A gauge type useful when Plugins like [syslog](./syslog.md), [http-logger](./http-logger.md), [tcp-logger](./tcp-logger.md), [udp-logger](./udp-logger.md), and [zipkin](./zipkin.md) use batch process to send data. Entries that hasn't been sent in batch process will be counted in the metrics.
-- Latency: Histogram of the request time per service in different dimensions.
+### Add Extra Labels for Metrics
- The available attributes are:
+The following example demonstrates how to add additional labels to metrics and use the [Nginx variable](https://nginx.org/en/docs/http/ngx_http_core_module.html) in label values.
- | Name | Description |
- |----------|-------------------------------------------------------------------------------------------------------------------------------------|
- | type | Value can be one of `apisix`, `upstream`, or `request`. This translates to latency caused by APISIX, Upstream, or both (their sum). |
- | route | `route_id` of the matched Route with request. Defaults to an empty string if the Routes don't match. |
- | service | `service_id` of the Route matching the request. If the Route does not have a `service_id` configured, it defaults to `$host`. |
- | consumer | `consumer_name` of the Consumer matching the request. Defaults to an empty string if it does not match. |
- | node | IP address of the Upstream node. |
+Currently, only the following metrics support extra labels:
-- Info: Information about the APISIX node.
-- Shared dict: The capacity and free space of all nginx.shared.DICT in APISIX.
+* apisix_http_status
+* apisix_http_latency
+* apisix_bandwidth
-- `apisix_upstream_status`: Health check result status of upstream nodes. A value of `1` represents healthy and `0` represents unhealthy.
+Include the following configurations in the [configuration based on environments](https://apisix.apache.org/docs/apisix/profile/) to add labels for metrics and reload APISIX for changes to take effect:
- The available attributes are:
+```yaml title="conf/config.yaml"
+plugin_attr:
+ prometheus: # Plugin: prometheus
+ metrics: # Create extra labels from the NGINX variables.
+ http_status:
+ extra_labels: # Set the extra labels for http_status metrics.
+ - upstream_addr: $upstream_addr # Add an extra upstream_addr label with value being the NGINX variable $upstream_addr.
+ - route_name: $route_name # Add an extra route_name label with value being the APISIX variable $route_name.
+```
- | Name | Description |
- |--------------|-------------------------------------------------------------------------------------------------------------------------------|
- | name | resource id where the upstream node is attached to, e.g. `/apisix/routes/1`, `/apisix/upstreams/1`. |
- | ip | ip address of the node. |
- | port | port number of the node. |
+Note that if you define a variable in the label value but it does not correspond to any existing [Nginx variable](https://nginx.org/en/docs/http/ngx_http_core_module.html), the label value will default to an empty string.
-Here are the original metrics from APISIX:
+Create a Route with the `prometheus` Plugin:
```shell
-curl http://127.0.0.1:9091/apisix/prometheus/metrics
+curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
+ -H "X-API-KEY: ${admin_key}" \
+ -d '{
+ "id": "prometheus-route",
+ "uri": "/get",
+ "name": "extra-label",
+ "plugins": {
+ "prometheus": {}
+ },
+ "upstream": {
+ "nodes": {
+ "httpbin.org:80": 1
+ }
+ }
+ }'
```
+Send a request to the Route to verify:
+
```shell
-# HELP apisix_bandwidth Total bandwidth in bytes consumed per service in Apisix
-# TYPE apisix_bandwidth counter
-apisix_bandwidth{type="egress",route="",service="",consumer="",node=""} 8417
-apisix_bandwidth{type="egress",route="1",service="",consumer="",node="127.0.0.1"} 1420
-apisix_bandwidth{type="egress",route="2",service="",consumer="",node="127.0.0.1"} 1420
-apisix_bandwidth{type="ingress",route="",service="",consumer="",node=""} 189
-apisix_bandwidth{type="ingress",route="1",service="",consumer="",node="127.0.0.1"} 332
-apisix_bandwidth{type="ingress",route="2",service="",consumer="",node="127.0.0.1"} 332
-# HELP apisix_etcd_modify_indexes Etcd modify index for APISIX keys
-# TYPE apisix_etcd_modify_indexes gauge
-apisix_etcd_modify_indexes{key="consumers"} 0
-apisix_etcd_modify_indexes{key="global_rules"} 0
-apisix_etcd_modify_indexes{key="max_modify_index"} 222
-apisix_etcd_modify_indexes{key="prev_index"} 35
-apisix_etcd_modify_indexes{key="protos"} 0
-apisix_etcd_modify_indexes{key="routes"} 222
-apisix_etcd_modify_indexes{key="services"} 0
-apisix_etcd_modify_indexes{key="ssls"} 0
-apisix_etcd_modify_indexes{key="stream_routes"} 0
-apisix_etcd_modify_indexes{key="upstreams"} 0
-apisix_etcd_modify_indexes{key="x_etcd_index"} 223
-# HELP apisix_batch_process_entries batch process remaining entries
-# TYPE apisix_batch_process_entries gauge
-apisix_batch_process_entries{name="http-logger",route_id="9",server_addr="127.0.0.1"} 1
-apisix_batch_process_entries{name="sls-logger",route_id="9",server_addr="127.0.0.1"} 1
-apisix_batch_process_entries{name="tcp-logger",route_id="9",server_addr="127.0.0.1"} 1
-apisix_batch_process_entries{name="udp-logger",route_id="9",server_addr="127.0.0.1"} 1
-apisix_batch_process_entries{name="sys-logger",route_id="9",server_addr="127.0.0.1"} 1
-apisix_batch_process_entries{name="zipkin_report",route_id="9",server_addr="127.0.0.1"} 1
-# HELP apisix_etcd_reachable Config server etcd reachable from Apisix, 0 is unreachable
-# TYPE apisix_etcd_reachable gauge
-apisix_etcd_reachable 1
-# HELP apisix_http_status HTTP status codes per service in Apisix
-# TYPE apisix_http_status counter
-apisix_http_status{code="200",route="1",matched_uri="/hello",matched_host="",service="",consumer="",node="127.0.0.1"} 4
-apisix_http_status{code="200",route="2",matched_uri="/world",matched_host="",service="",consumer="",node="127.0.0.1"} 4
-apisix_http_status{code="404",route="",matched_uri="",matched_host="",service="",consumer="",node=""} 1
-# HELP apisix_http_requests_total The total number of client requests
-# TYPE apisix_http_requests_total gauge
-apisix_http_requests_total 1191780
-# HELP apisix_nginx_http_current_connections Number of HTTP connections
-# TYPE apisix_nginx_http_current_connections gauge
-apisix_nginx_http_current_connections{state="accepted"} 11994
-apisix_nginx_http_current_connections{state="active"} 2
-apisix_nginx_http_current_connections{state="handled"} 11994
-apisix_nginx_http_current_connections{state="reading"} 0
-apisix_nginx_http_current_connections{state="waiting"} 1
-apisix_nginx_http_current_connections{state="writing"} 1
-# HELP apisix_nginx_metric_errors_total Number of nginx-lua-prometheus errors
-# TYPE apisix_nginx_metric_errors_total counter
-apisix_nginx_metric_errors_total 0
-# HELP apisix_http_latency HTTP request latency in milliseconds per service in APISIX
-# TYPE apisix_http_latency histogram
-apisix_http_latency_bucket{type="apisix",route="1",service="",consumer="",node="127.0.0.1",le="1"} 1
-apisix_http_latency_bucket{type="apisix",route="1",service="",consumer="",node="127.0.0.1",le="2"} 1
-apisix_http_latency_bucket{type="request",route="1",service="",consumer="",node="127.0.0.1",le="1"} 1
-apisix_http_latency_bucket{type="request",route="1",service="",consumer="",node="127.0.0.1",le="2"} 1
-apisix_http_latency_bucket{type="upstream",route="1",service="",consumer="",node="127.0.0.1",le="1"} 1
-apisix_http_latency_bucket{type="upstream",route="1",service="",consumer="",node="127.0.0.1",le="2"} 1
-...
-# HELP apisix_node_info Info of APISIX node
-# TYPE apisix_node_info gauge
-apisix_node_info{hostname="desktop-2022q8f-wsl"} 1
-# HELP apisix_shared_dict_capacity_bytes The capacity of each nginx shared DICT since APISIX start
-# TYPE apisix_shared_dict_capacity_bytes gauge
-apisix_shared_dict_capacity_bytes{name="access-tokens"} 1048576
-apisix_shared_dict_capacity_bytes{name="balancer-ewma"} 10485760
-apisix_shared_dict_capacity_bytes{name="balancer-ewma-last-touched-at"} 10485760
-apisix_shared_dict_capacity_bytes{name="balancer-ewma-locks"} 10485760
-apisix_shared_dict_capacity_bytes{name="discovery"} 1048576
-apisix_shared_dict_capacity_bytes{name="etcd-cluster-health-check"} 10485760
-...
-# HELP apisix_shared_dict_free_space_bytes The free space of each nginx shared DICT since APISIX start
-# TYPE apisix_shared_dict_free_space_bytes gauge
-apisix_shared_dict_free_space_bytes{name="access-tokens"} 1032192
-apisix_shared_dict_free_space_bytes{name="balancer-ewma"} 10412032
-apisix_shared_dict_free_space_bytes{name="balancer-ewma-last-touched-at"} 10412032
-apisix_shared_dict_free_space_bytes{name="balancer-ewma-locks"} 10412032
-apisix_shared_dict_free_space_bytes{name="discovery"} 1032192
-apisix_shared_dict_free_space_bytes{name="etcd-cluster-health-check"} 10412032
-...
-# HELP apisix_upstream_status Upstream status from health check
-# TYPE apisix_upstream_status gauge
-apisix_upstream_status{name="/apisix/routes/1",ip="100.24.156.8",port="80"} 0
-apisix_upstream_status{name="/apisix/routes/1",ip="52.86.68.46",port="80"} 1
+curl -i "http://127.0.0.1:9080/get"
```
-## Delete Plugin
+You should see an `HTTP/1.1 200 OK` response.
-To remove the `prometheus` Plugin, you can delete the corresponding JSON configuration from the Plugin configuration. APISIX will automatically reload and you do not have to restart for this to take effect.
+Send a request to the APISIX Prometheus metrics endpoint:
```shell
-curl http://127.0.0.1:9180/apisix/admin/routes/1 -H "X-API-KEY: $admin_key" -X PUT -d '
-{
- "uri": "/hello",
- "plugins": {},
- "upstream": {
- "type": "roundrobin",
- "nodes": {
- "127.0.0.1:80": 1
- }
- }
-}'
+curl "http://127.0.0.1:9091/apisix/prometheus/metrics"
```
-## How to enable it for TCP/UDP
+You should see an output similar to the following:
-:::info IMPORTANT
-
-This feature requires APISIX to run on [APISIX-Runtime](../FAQ.md#how-do-i-build-the-apisix-runtime-environment?).
+```text
+# HELP apisix_http_status HTTP status codes per Service in APISIX
+# TYPE apisix_http_status counter
+apisix_http_status{code="200",route="1",matched_uri="/get",matched_host="",service="",consumer="",node="54.237.103.220",upstream_addr="54.237.103.220:80",route_name="extra-label"} 1
+```
-:::
+### Monitor TCP/UDP Traffic with Prometheus
-We can also enable `prometheus` to collect metrics for TCP/UDP.
+The following example demonstrates how to collect TCP/UDP traffic metrics in APISIX.
-First of all, ensure `prometheus` plugin is in your configuration file (`conf/config.yaml`):
+Include the following configurations in the [configuration based on environments](https://apisix.apache.org/docs/apisix/profile/) to enable stream proxy and `prometheus` Plugin for stream proxy. Reload APISIX for changes to take effect:
Review Comment:
```suggestion
Include the following configurations in `config.yaml` to enable stream proxy and enable `prometheus` Plugin for stream proxy. Reload APISIX for changes to take effect:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs: improve prometheus plugin docs [apisix]
Posted by "Yilialinn (via GitHub)" <gi...@apache.org>.
Yilialinn commented on code in PR #11874:
URL: https://github.com/apache/apisix/pull/11874#discussion_r1899402049
##########
docs/en/latest/plugins/prometheus.md:
##########
@@ -27,438 +27,423 @@ description: This document contains information about the Apache APISIX promethe
#
-->
-## Description
-
-The `prometheus` Plugin exports metrics in [Prometheus exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/#exposition-formats).
-
-## Attributes
-
-| Name | Type | Required | Default | Description |
-| ----------- | ------- | -------- | ------- | --------------------------------------------------------------------------------- |
-| prefer_name | boolean | False | false | When set to `true`, prints Route/Service name instead of ID in Prometheus metric. |
+<head>
+ <link rel="canonical" href="https://docs.api7.ai/hub/prometheus" />
+</head>
-### Specifying `export_uri`
+## Description
-You can change the default export URI by configuring the `export_uri` attribute under `plugin_attr` in your configuration file (`conf/config.yaml`).
+The `prometheus` plugin provides the capability to integrate APISIX with [Prometheus](https://prometheus.io).
-| Name | Type | Default | Description |
-| ---------- | ------ | ---------------------------- | ------------------------------------- |
-| export_uri | string | "/apisix/prometheus/metrics" | URI to export the Prometheus metrics. |
+After enabling the plugin, APISIX will start collecting relevant metrics, such as API requests and latencies, and exporting them in a [text-based exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/#exposition-formats) to Prometheus. You can then create event monitoring and alerting in Prometheus to monitor the health of your API gateway and APIs.
-Here is a configuration example:
+## Attributes
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- export_uri: /apisix/metrics
+There are different types of metrics in Prometheus. To understand their differences, see [metrics types](https://prometheus.io/docs/concepts/metric_types/).
+
+The following metrics are exported by the `prometheus` plugin by default. See [get APISIX metrics](#get-apisix-metrics) for an example. Note that some metrics, such as `apisix_batch_process_entries`, are not readily visible if there are no data.
+
+
+| Name | Type | Description |
+| ------------------------------ | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| apisix_bandwidth | counter | Total amount of traffic flowing through APISIX in bytes. |
+| apisix_etcd_modify_indexes | gauge | Number of changes to etcd by APISIX keys. |
+| apisix_batch_process_entries | gauge | Number of remaining entries in a batch when sending data in batches, such as with `http logger`, and other logging plugins. |
+| apisix_etcd_reachable | gauge | Whether APISIX can reach etcd. A value of `1` represents reachable and `0` represents unreachable. |
+| apisix_http_status | counter | HTTP status codes returned from upstream services. |
+| apisix_http_requests_total | gauge | Number of HTTP requests from clients. |
+| apisix_nginx_http_current_connections | gauge | Number of current connections with clients. |
+| apisix_nginx_metric_errors_total | counter | Total number of `nginx-lua-prometheus` errors. |
+| apisix_http_latency | histogram | HTTP request latency in milliseconds. |
+| apisix_node_info | gauge | Information of the APISIX node, such as host name. |
+| apisix_shared_dict_capacity_bytes | gauge | The total capacity of an [NGINX shared dictionary](https://github.com/openresty/lua-nginx-module#ngxshareddict). |
+| apisix_shared_dict_free_space_bytes | gauge | The remaining space in an [NGINX shared dictionary](https://github.com/openresty/lua-nginx-module#ngxshareddict). |
+| apisix_upstream_status | gauge | Health check status of upstream nodes, available if health checks are configured on the upstream. A value of `1` represents healthy and `0` represents unhealthy. |
+| apisix_stream_connection_total | counter | Total number of connections handled per stream route. |
+
+## Labels
+
+[Labels](https://prometheus.io/docs/practices/naming/#labels) are attributes of metrics that are used to differentiate metrics.
+
+For example, the `apisix_http_status` metric can be labeled with `route` information to identify which route the HTTP status originates from.
+
+The following are labels for a non-exhaustive list of APISIX metrics and their descriptions.
+
+### Labels for `apisix_http_status`
+
+The following labels are used to differentiate `apisix_http_status` metrics.
+
+| Name | Description |
+| ------------ | ----------------------------------------------------------------------------------------------------------------------------- |
+| code | HTTP response code returned by the upstream node. |
+| route | ID of the route that the HTTP status originates from when `prefer_name` is `false` (default), and name of the route when `prefer_name` to `true`. Default to an empty string if a request does not match any route. |
+| route_id | Available only in Enterprise. ID of the route that the HTTP status originates from regardless of the `prefer_name` setting. |
+| matched_uri | URI of the route that matches the request. Default to an empty string if a request does not match any route. |
+| matched_host | Host of the route that matches the request. Default to an empty string if a request does not match any route, or host is not configured on the route. |
+| service | ID of the service that the HTTP status originates from when `prefer_name` is `false` (default), and name of the service when `prefer_name` to `true`. Default to the configured value of host on the route if the matched route does not belong to any service. |
+| service_id | Available only in Enterprise. ID of the service that the HTTP status originates from regardless of the `prefer_name` setting. |
+| consumer | Name of the consumer associated with a request. Default to an empty string if no consumer is associated with the request. |
+| node | IP address of the upstream node. |
+| gateway_group_id | Available only in Enterprise. ID of the gateway group that the HTTP status originates from. |
+| instance_id | Available only in Enterprise. ID of the gateway instance that the HTTP status originates from. |
+| api_product_id | Available only in Enterprise. Product ID that the HTTP status originates from. |
+
+### Labels for `apisix_bandwidth`
+
+The following labels are used to differentiate `apisix_bandwidth` metrics.
+
+| Name | Description |
+| ---------- | ----------------------------------------------------------------------------------------------------------------------------- |
+| type | Type of traffic, `egress` or `ingress`. |
+| route | ID of the route that bandwidth corresponds to when `prefer_name` is `false` (default), and name of the route when `prefer_name` to `true`. Default to an empty string if a request does not match any route. |
+| route_id | Available only in Enterprise. ID of the route that bandwidth corresponds to regardless of the `prefer_name` setting. |
+| service | ID of the service that bandwidth corresponds to when `prefer_name` is `false` (default), and name of the service when `prefer_name` to `true`. Default to the configured value of host on the route if the matched route does not belong to any service. |
+| service_id | Available only in Enterprise. ID of the service that bandwidth corresponds to regardless of the `prefer_name` setting. |
+| consumer | Name of the consumer associated with a request. Default to an empty string if no consumer is associated with the request. |
+| node | IP address of the upstream node. |
+| gateway_group_id | Available only in Enterprise. ID of the gateway group that bandwidth corresponds to. |
+| instance_id | Available only in Enterprise. ID of the gateway instance that bandwidth corresponds to. |
+| api_product_id | Available only in Enterprise. Product ID that bandwidth corresponds to. |
+
+### Labels for `apisix_http_latency`
+
+The following labels are used to differentiate `apisix_http_latency` metrics.
+
+| Name | Description |
+| ---------- | ----------------------------------------------------------------------------------------------------------------------------------- |
+| type | Type of latencies. See [latency types](#latency-types) for details. |
+| route | ID of the route that latencies correspond to when `prefer_name` is `false` (default), and name of the route when `prefer_name` to `true`. Default to an empty string if a request does not match any route. |
+| route_id | Available only in Enterprise. ID of the route that latencies correspond to regardless of the `prefer_name` setting. |
+| service | ID of the service that latencies correspond to when `prefer_name` is `false` (default), and name of the service when `prefer_name` to `true`. Default to the configured value of host on the route if the matched route does not belong to any service. |
+| service_id | Available only in Enterprise. ID of the service that latencies correspond to regardless of the `prefer_name` setting. |
+| consumer | Name of the consumer associated with latencies. Default to an empty string if no consumer is associated with the request. |
+| node | IP address of the upstream node associated with latencies. |
+| gateway_group_id | Available only in Enterprise. ID of the gateway group that latencies correspond to. |
+| instance_id | Available only in Enterprise. ID of the gateway instance that latencies correspond to. |
+| api_product_id | Available only in Enterprise. Product ID that latencies correspond to. |
+
+#### Latency Types
+
+`apisix_http_latency` can be labeled with one of the three types:
+
+* `request` represents the time elapsed between the first byte was read from the client and the log write after the last byte was sent to the client.
+
+* `upstream` represents the time elapsed waiting on responses from the upstream service.
+
+* `apisix` represents the difference between the `request` latency and `upstream` latency.
+
+In other words, the APISIX latency is not only attributed to the Lua processing. It should be understood as follows:
+
+```text
+APISIX latency
+ = downstream request time - upstream response time
+ = downstream traffic latency + NGINX latency
```
-### Specifying `metrics`
-
-For http request related metrics, you could specify extra labels, which match the APISIX variables.
+### Labels for `apisix_upstream_status`
-If you specify label for nonexist APISIX variable, the label value would be "".
+The following labels are used to differentiate `apisix_upstream_status` metrics.
-Currently, only below metrics are supported:
+| Name | Description |
+| ---------- | --------------------------------------------------------------------------------------------------- |
+| name | Resource ID corresponding to the upstream configured with health checks, such as `/apisix/routes/1` and `/apisix/upstreams/1`. |
+| ip | IP address of the upstream node. |
+| port | Port number of the node. |
-* http_status
-* http_latency
-* bandwidth
+## Examples
-Here is a configuration example:
-
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- metrics:
- http_status:
- extra_labels:
- - upstream_addr: $upstream_addr
- - upstream_status: $upstream_status
-```
+The examples below demonstrate how you can work with the `prometheus` plugin for different scenarios.
-### Specifying `default_buckets`
+### Get APISIX Metrics
-`DEFAULT_BUCKETS` is the default value for bucket array in `http_latency` metrics.
+The following example demonstrates how you can get metrics from APISIX.
-You can change the `DEFAULT_BUCKETS` by configuring `default_buckets` attribute in you configuration file.
+The default Prometheus metrics endpoint and other Prometheus related configurations can be found in the [static configuration](#static-configurations). If you would like to customize these configuration, see [configuration files](/apisix/reference/configuration-files#configyaml-and-configyamlexample).
Review Comment:
no related content of static-configurations @kayx23
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs: improve prometheus plugin docs [apisix]
Posted by "kayx23 (via GitHub)" <gi...@apache.org>.
kayx23 commented on code in PR #11874:
URL: https://github.com/apache/apisix/pull/11874#discussion_r1899957740
##########
docs/en/latest/plugins/prometheus.md:
##########
@@ -5,7 +5,7 @@ keywords:
- API Gateway
- Plugin
- Prometheus
-description: This document contains information about the Apache APISIX prometheus Plugin.
+description: The prometheus plugin provides the capability to integrate APISIX with Prometheus for metric collection and continuous monitoring.
Review Comment:
all `plugin` needs to be capitalized - `Plugin`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs: improve prometheus plugin docs [apisix]
Posted by "Yilialinn (via GitHub)" <gi...@apache.org>.
Yilialinn commented on code in PR #11874:
URL: https://github.com/apache/apisix/pull/11874#discussion_r1900603637
##########
docs/en/latest/plugins/prometheus.md:
##########
@@ -27,439 +27,471 @@ description: This document contains information about the Apache APISIX promethe
#
-->
-## Description
-
-The `prometheus` Plugin exports metrics in [Prometheus exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/#exposition-formats).
+<head>
+ <link rel="canonical" href="https://docs.api7.ai/hub/prometheus" />
+</head>
-## Attributes
+## Description
-| Name | Type | Required | Default | Description |
-| ----------- | ------- | -------- | ------- | --------------------------------------------------------------------------------- |
-| prefer_name | boolean | False | false | When set to `true`, prints Route/Service name instead of ID in Prometheus metric. |
+The `prometheus` Plugin provides the capability to integrate APISIX with [Prometheus](https://prometheus.io).
-### Specifying `export_uri`
+After enabling the Plugin, APISIX will start collecting relevant metrics, such as API requests and latencies, and exporting them in a [text-based exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/#exposition-formats) to Prometheus. You can then create event monitoring and alerting in Prometheus to monitor the health of your API gateway and APIs.
-You can change the default export URI by configuring the `export_uri` attribute under `plugin_attr` in your configuration file (`conf/config.yaml`).
+## Static Configurations
-| Name | Type | Default | Description |
-| ---------- | ------ | ---------------------------- | ------------------------------------- |
-| export_uri | string | "/apisix/prometheus/metrics" | URI to export the Prometheus metrics. |
+By default, `prometheus` configurations are pre-configured in the [default configuration](https://github.com/apache/apisix/blob/master/apisix/cli/config.lua).
-Here is a configuration example:
+To customize these values, add the corresponding configurations to `config.yaml`. For example:
-```yaml title="conf/config.yaml"
+```yaml
plugin_attr:
- prometheus:
- export_uri: /apisix/metrics
+ prometheus: # Plugin: prometheus attributes
+ export_uri: /apisix/prometheus/metrics # Set the URI for the Prometheus metrics endpoint.
+ metric_prefix: apisix_ # Set the prefix for Prometheus metrics generated by APISIX.
+ enable_export_server: true # Enable the Prometheus export server.
+ export_addr: # Set the address for the Prometheus export server.
+ ip: 127.0.0.1 # Set the IP.
+ port: 9091 # Set the port.
+ # metrics: # Create extra labels for metrics.
+ # http_status: # These metrics will be prefixed with `apisix_`.
+ # extra_labels: # Set the extra labels for http_status metrics.
+ # - upstream_addr: $upstream_addr
+ # - status: $upstream_status
+ # expire: 0 # The expiration time of metrics in seconds.
+ # 0 means the metrics will not expire.
+ # http_latency:
+ # extra_labels: # Set the extra labels for http_latency metrics.
+ # - upstream_addr: $upstream_addr
+ # expire: 0 # The expiration time of metrics in seconds.
+ # 0 means the metrics will not expire.
+ # bandwidth:
+ # extra_labels: # Set the extra labels for bandwidth metrics.
+ # - upstream_addr: $upstream_addr
+ # expire: 0 # The expiration time of metrics in seconds.
+ # 0 means the metrics will not expire.
+ # default_buckets: # Set the default buckets for the `http_latency` metrics histogram.
+ # - 10
+ # - 50
+ # - 100
+ # - 200
+ # - 500
+ # - 1000
+ # - 2000
+ # - 5000
+ # - 10000
+ # - 30000
+ # - 60000
+ # - 500
```
-### Specifying `metrics`
+You can use the [Nginx variable](https://nginx.org/en/docs/http/ngx_http_core_module.html) to create `extra_labels`. See [add extra labels](/hub/prometheus#add-extra-labels-for-metrics).
-For http request related metrics, you could specify extra labels, which match the APISIX variables.
+Reload APISIX for changes to take effect.
-If you specify label for nonexist APISIX variable, the label value would be "".
+## Attribute
-Currently, only below metrics are supported:
+| Name | Type | Required | Default | Valid values | Description |
+| ------------- | ------- | -------- | ------- | ------------ | ------------------------------------------ |
+| `prefer_name` | boolean | | False | | If true, export Route/Service name instead of their ID in Prometheus metrics. |
-* http_status
-* http_latency
-* bandwidth
+## Metrics
-Here is a configuration example:
+There are different types of metrics in Prometheus. To understand their differences, see [metrics types](https://prometheus.io/docs/concepts/metric_types/).
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- metrics:
- http_status:
- extra_labels:
- - upstream_addr: $upstream_addr
- - upstream_status: $upstream_status
-```
+The following metrics are exported by the `prometheus` Plugin by default. See [get APISIX metrics](#get-apisix-metrics) for an example. Note that some metrics, such as `apisix_batch_process_entries`, are not readily visible if there are no data.
-### Specifying `default_buckets`
+| Name | Type | Description |
+| ------------------------------ | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| apisix_bandwidth | counter | Total amount of traffic flowing through APISIX in bytes. |
+| apisix_etcd_modify_indexes | gauge | Number of changes to etcd by APISIX keys. |
+| apisix_batch_process_entries | gauge | Number of remaining entries in a batch when sending data in batches, such as with `http logger`, and other logging Plugins. |
+| apisix_etcd_reachable | gauge | Whether APISIX can reach etcd. A value of `1` represents reachable and `0` represents unreachable. |
+| apisix_http_status | counter | HTTP status codes returned from upstream Services. |
+| apisix_http_requests_total | gauge | Number of HTTP requests from clients. |
+| apisix_nginx_http_current_connections | gauge | Number of current connections with clients. |
+| apisix_nginx_metric_errors_total | counter | Total number of `nginx-lua-prometheus` errors. |
+| apisix_http_latency | histogram | HTTP request latency in milliseconds. |
+| apisix_node_info | gauge | Information of the APISIX node, such as host name. |
+| apisix_shared_dict_capacity_bytes | gauge | The total capacity of an [NGINX shared dictionary](https://github.com/openresty/lua-nginx-module#ngxshareddict). |
+| apisix_shared_dict_free_space_bytes | gauge | The remaining space in an [NGINX shared dictionary](https://github.com/openresty/lua-nginx-module#ngxshareddict). |
+| apisix_upstream_status | gauge | Health check status of upstream nodes, available if health checks are configured on the upstream. A value of `1` represents healthy and `0` represents unhealthy. |
+| apisix_stream_connection_total | counter | Total number of connections handled per stream Route. |
Review Comment:
Fixed those in sentences
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs: improve prometheus plugin docs [apisix]
Posted by "Yilialinn (via GitHub)" <gi...@apache.org>.
Yilialinn commented on code in PR #11874:
URL: https://github.com/apache/apisix/pull/11874#discussion_r1899402049
##########
docs/en/latest/plugins/prometheus.md:
##########
@@ -27,438 +27,423 @@ description: This document contains information about the Apache APISIX promethe
#
-->
-## Description
-
-The `prometheus` Plugin exports metrics in [Prometheus exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/#exposition-formats).
-
-## Attributes
-
-| Name | Type | Required | Default | Description |
-| ----------- | ------- | -------- | ------- | --------------------------------------------------------------------------------- |
-| prefer_name | boolean | False | false | When set to `true`, prints Route/Service name instead of ID in Prometheus metric. |
+<head>
+ <link rel="canonical" href="https://docs.api7.ai/hub/prometheus" />
+</head>
-### Specifying `export_uri`
+## Description
-You can change the default export URI by configuring the `export_uri` attribute under `plugin_attr` in your configuration file (`conf/config.yaml`).
+The `prometheus` plugin provides the capability to integrate APISIX with [Prometheus](https://prometheus.io).
-| Name | Type | Default | Description |
-| ---------- | ------ | ---------------------------- | ------------------------------------- |
-| export_uri | string | "/apisix/prometheus/metrics" | URI to export the Prometheus metrics. |
+After enabling the plugin, APISIX will start collecting relevant metrics, such as API requests and latencies, and exporting them in a [text-based exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/#exposition-formats) to Prometheus. You can then create event monitoring and alerting in Prometheus to monitor the health of your API gateway and APIs.
-Here is a configuration example:
+## Attributes
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- export_uri: /apisix/metrics
+There are different types of metrics in Prometheus. To understand their differences, see [metrics types](https://prometheus.io/docs/concepts/metric_types/).
+
+The following metrics are exported by the `prometheus` plugin by default. See [get APISIX metrics](#get-apisix-metrics) for an example. Note that some metrics, such as `apisix_batch_process_entries`, are not readily visible if there are no data.
+
+
+| Name | Type | Description |
+| ------------------------------ | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| apisix_bandwidth | counter | Total amount of traffic flowing through APISIX in bytes. |
+| apisix_etcd_modify_indexes | gauge | Number of changes to etcd by APISIX keys. |
+| apisix_batch_process_entries | gauge | Number of remaining entries in a batch when sending data in batches, such as with `http logger`, and other logging plugins. |
+| apisix_etcd_reachable | gauge | Whether APISIX can reach etcd. A value of `1` represents reachable and `0` represents unreachable. |
+| apisix_http_status | counter | HTTP status codes returned from upstream services. |
+| apisix_http_requests_total | gauge | Number of HTTP requests from clients. |
+| apisix_nginx_http_current_connections | gauge | Number of current connections with clients. |
+| apisix_nginx_metric_errors_total | counter | Total number of `nginx-lua-prometheus` errors. |
+| apisix_http_latency | histogram | HTTP request latency in milliseconds. |
+| apisix_node_info | gauge | Information of the APISIX node, such as host name. |
+| apisix_shared_dict_capacity_bytes | gauge | The total capacity of an [NGINX shared dictionary](https://github.com/openresty/lua-nginx-module#ngxshareddict). |
+| apisix_shared_dict_free_space_bytes | gauge | The remaining space in an [NGINX shared dictionary](https://github.com/openresty/lua-nginx-module#ngxshareddict). |
+| apisix_upstream_status | gauge | Health check status of upstream nodes, available if health checks are configured on the upstream. A value of `1` represents healthy and `0` represents unhealthy. |
+| apisix_stream_connection_total | counter | Total number of connections handled per stream route. |
+
+## Labels
+
+[Labels](https://prometheus.io/docs/practices/naming/#labels) are attributes of metrics that are used to differentiate metrics.
+
+For example, the `apisix_http_status` metric can be labeled with `route` information to identify which route the HTTP status originates from.
+
+The following are labels for a non-exhaustive list of APISIX metrics and their descriptions.
+
+### Labels for `apisix_http_status`
+
+The following labels are used to differentiate `apisix_http_status` metrics.
+
+| Name | Description |
+| ------------ | ----------------------------------------------------------------------------------------------------------------------------- |
+| code | HTTP response code returned by the upstream node. |
+| route | ID of the route that the HTTP status originates from when `prefer_name` is `false` (default), and name of the route when `prefer_name` to `true`. Default to an empty string if a request does not match any route. |
+| route_id | Available only in Enterprise. ID of the route that the HTTP status originates from regardless of the `prefer_name` setting. |
+| matched_uri | URI of the route that matches the request. Default to an empty string if a request does not match any route. |
+| matched_host | Host of the route that matches the request. Default to an empty string if a request does not match any route, or host is not configured on the route. |
+| service | ID of the service that the HTTP status originates from when `prefer_name` is `false` (default), and name of the service when `prefer_name` to `true`. Default to the configured value of host on the route if the matched route does not belong to any service. |
+| service_id | Available only in Enterprise. ID of the service that the HTTP status originates from regardless of the `prefer_name` setting. |
+| consumer | Name of the consumer associated with a request. Default to an empty string if no consumer is associated with the request. |
+| node | IP address of the upstream node. |
+| gateway_group_id | Available only in Enterprise. ID of the gateway group that the HTTP status originates from. |
+| instance_id | Available only in Enterprise. ID of the gateway instance that the HTTP status originates from. |
+| api_product_id | Available only in Enterprise. Product ID that the HTTP status originates from. |
+
+### Labels for `apisix_bandwidth`
+
+The following labels are used to differentiate `apisix_bandwidth` metrics.
+
+| Name | Description |
+| ---------- | ----------------------------------------------------------------------------------------------------------------------------- |
+| type | Type of traffic, `egress` or `ingress`. |
+| route | ID of the route that bandwidth corresponds to when `prefer_name` is `false` (default), and name of the route when `prefer_name` to `true`. Default to an empty string if a request does not match any route. |
+| route_id | Available only in Enterprise. ID of the route that bandwidth corresponds to regardless of the `prefer_name` setting. |
+| service | ID of the service that bandwidth corresponds to when `prefer_name` is `false` (default), and name of the service when `prefer_name` to `true`. Default to the configured value of host on the route if the matched route does not belong to any service. |
+| service_id | Available only in Enterprise. ID of the service that bandwidth corresponds to regardless of the `prefer_name` setting. |
+| consumer | Name of the consumer associated with a request. Default to an empty string if no consumer is associated with the request. |
+| node | IP address of the upstream node. |
+| gateway_group_id | Available only in Enterprise. ID of the gateway group that bandwidth corresponds to. |
+| instance_id | Available only in Enterprise. ID of the gateway instance that bandwidth corresponds to. |
+| api_product_id | Available only in Enterprise. Product ID that bandwidth corresponds to. |
+
+### Labels for `apisix_http_latency`
+
+The following labels are used to differentiate `apisix_http_latency` metrics.
+
+| Name | Description |
+| ---------- | ----------------------------------------------------------------------------------------------------------------------------------- |
+| type | Type of latencies. See [latency types](#latency-types) for details. |
+| route | ID of the route that latencies correspond to when `prefer_name` is `false` (default), and name of the route when `prefer_name` to `true`. Default to an empty string if a request does not match any route. |
+| route_id | Available only in Enterprise. ID of the route that latencies correspond to regardless of the `prefer_name` setting. |
+| service | ID of the service that latencies correspond to when `prefer_name` is `false` (default), and name of the service when `prefer_name` to `true`. Default to the configured value of host on the route if the matched route does not belong to any service. |
+| service_id | Available only in Enterprise. ID of the service that latencies correspond to regardless of the `prefer_name` setting. |
+| consumer | Name of the consumer associated with latencies. Default to an empty string if no consumer is associated with the request. |
+| node | IP address of the upstream node associated with latencies. |
+| gateway_group_id | Available only in Enterprise. ID of the gateway group that latencies correspond to. |
+| instance_id | Available only in Enterprise. ID of the gateway instance that latencies correspond to. |
+| api_product_id | Available only in Enterprise. Product ID that latencies correspond to. |
+
+#### Latency Types
+
+`apisix_http_latency` can be labeled with one of the three types:
+
+* `request` represents the time elapsed between the first byte was read from the client and the log write after the last byte was sent to the client.
+
+* `upstream` represents the time elapsed waiting on responses from the upstream service.
+
+* `apisix` represents the difference between the `request` latency and `upstream` latency.
+
+In other words, the APISIX latency is not only attributed to the Lua processing. It should be understood as follows:
+
+```text
+APISIX latency
+ = downstream request time - upstream response time
+ = downstream traffic latency + NGINX latency
```
-### Specifying `metrics`
-
-For http request related metrics, you could specify extra labels, which match the APISIX variables.
+### Labels for `apisix_upstream_status`
-If you specify label for nonexist APISIX variable, the label value would be "".
+The following labels are used to differentiate `apisix_upstream_status` metrics.
-Currently, only below metrics are supported:
+| Name | Description |
+| ---------- | --------------------------------------------------------------------------------------------------- |
+| name | Resource ID corresponding to the upstream configured with health checks, such as `/apisix/routes/1` and `/apisix/upstreams/1`. |
+| ip | IP address of the upstream node. |
+| port | Port number of the node. |
-* http_status
-* http_latency
-* bandwidth
+## Examples
-Here is a configuration example:
-
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- metrics:
- http_status:
- extra_labels:
- - upstream_addr: $upstream_addr
- - upstream_status: $upstream_status
-```
+The examples below demonstrate how you can work with the `prometheus` plugin for different scenarios.
-### Specifying `default_buckets`
+### Get APISIX Metrics
-`DEFAULT_BUCKETS` is the default value for bucket array in `http_latency` metrics.
+The following example demonstrates how you can get metrics from APISIX.
-You can change the `DEFAULT_BUCKETS` by configuring `default_buckets` attribute in you configuration file.
+The default Prometheus metrics endpoint and other Prometheus related configurations can be found in the [static configuration](#static-configurations). If you would like to customize these configuration, see [configuration files](/apisix/reference/configuration-files#configyaml-and-configyamlexample).
Review Comment:
no related content about static-configurations
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs: improve prometheus plugin docs [apisix]
Posted by "Yilialinn (via GitHub)" <gi...@apache.org>.
Yilialinn commented on code in PR #11874:
URL: https://github.com/apache/apisix/pull/11874#discussion_r1899402049
##########
docs/en/latest/plugins/prometheus.md:
##########
@@ -27,438 +27,423 @@ description: This document contains information about the Apache APISIX promethe
#
-->
-## Description
-
-The `prometheus` Plugin exports metrics in [Prometheus exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/#exposition-formats).
-
-## Attributes
-
-| Name | Type | Required | Default | Description |
-| ----------- | ------- | -------- | ------- | --------------------------------------------------------------------------------- |
-| prefer_name | boolean | False | false | When set to `true`, prints Route/Service name instead of ID in Prometheus metric. |
+<head>
+ <link rel="canonical" href="https://docs.api7.ai/hub/prometheus" />
+</head>
-### Specifying `export_uri`
+## Description
-You can change the default export URI by configuring the `export_uri` attribute under `plugin_attr` in your configuration file (`conf/config.yaml`).
+The `prometheus` plugin provides the capability to integrate APISIX with [Prometheus](https://prometheus.io).
-| Name | Type | Default | Description |
-| ---------- | ------ | ---------------------------- | ------------------------------------- |
-| export_uri | string | "/apisix/prometheus/metrics" | URI to export the Prometheus metrics. |
+After enabling the plugin, APISIX will start collecting relevant metrics, such as API requests and latencies, and exporting them in a [text-based exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/#exposition-formats) to Prometheus. You can then create event monitoring and alerting in Prometheus to monitor the health of your API gateway and APIs.
-Here is a configuration example:
+## Attributes
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- export_uri: /apisix/metrics
+There are different types of metrics in Prometheus. To understand their differences, see [metrics types](https://prometheus.io/docs/concepts/metric_types/).
+
+The following metrics are exported by the `prometheus` plugin by default. See [get APISIX metrics](#get-apisix-metrics) for an example. Note that some metrics, such as `apisix_batch_process_entries`, are not readily visible if there are no data.
+
+
+| Name | Type | Description |
+| ------------------------------ | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| apisix_bandwidth | counter | Total amount of traffic flowing through APISIX in bytes. |
+| apisix_etcd_modify_indexes | gauge | Number of changes to etcd by APISIX keys. |
+| apisix_batch_process_entries | gauge | Number of remaining entries in a batch when sending data in batches, such as with `http logger`, and other logging plugins. |
+| apisix_etcd_reachable | gauge | Whether APISIX can reach etcd. A value of `1` represents reachable and `0` represents unreachable. |
+| apisix_http_status | counter | HTTP status codes returned from upstream services. |
+| apisix_http_requests_total | gauge | Number of HTTP requests from clients. |
+| apisix_nginx_http_current_connections | gauge | Number of current connections with clients. |
+| apisix_nginx_metric_errors_total | counter | Total number of `nginx-lua-prometheus` errors. |
+| apisix_http_latency | histogram | HTTP request latency in milliseconds. |
+| apisix_node_info | gauge | Information of the APISIX node, such as host name. |
+| apisix_shared_dict_capacity_bytes | gauge | The total capacity of an [NGINX shared dictionary](https://github.com/openresty/lua-nginx-module#ngxshareddict). |
+| apisix_shared_dict_free_space_bytes | gauge | The remaining space in an [NGINX shared dictionary](https://github.com/openresty/lua-nginx-module#ngxshareddict). |
+| apisix_upstream_status | gauge | Health check status of upstream nodes, available if health checks are configured on the upstream. A value of `1` represents healthy and `0` represents unhealthy. |
+| apisix_stream_connection_total | counter | Total number of connections handled per stream route. |
+
+## Labels
+
+[Labels](https://prometheus.io/docs/practices/naming/#labels) are attributes of metrics that are used to differentiate metrics.
+
+For example, the `apisix_http_status` metric can be labeled with `route` information to identify which route the HTTP status originates from.
+
+The following are labels for a non-exhaustive list of APISIX metrics and their descriptions.
+
+### Labels for `apisix_http_status`
+
+The following labels are used to differentiate `apisix_http_status` metrics.
+
+| Name | Description |
+| ------------ | ----------------------------------------------------------------------------------------------------------------------------- |
+| code | HTTP response code returned by the upstream node. |
+| route | ID of the route that the HTTP status originates from when `prefer_name` is `false` (default), and name of the route when `prefer_name` to `true`. Default to an empty string if a request does not match any route. |
+| route_id | Available only in Enterprise. ID of the route that the HTTP status originates from regardless of the `prefer_name` setting. |
+| matched_uri | URI of the route that matches the request. Default to an empty string if a request does not match any route. |
+| matched_host | Host of the route that matches the request. Default to an empty string if a request does not match any route, or host is not configured on the route. |
+| service | ID of the service that the HTTP status originates from when `prefer_name` is `false` (default), and name of the service when `prefer_name` to `true`. Default to the configured value of host on the route if the matched route does not belong to any service. |
+| service_id | Available only in Enterprise. ID of the service that the HTTP status originates from regardless of the `prefer_name` setting. |
+| consumer | Name of the consumer associated with a request. Default to an empty string if no consumer is associated with the request. |
+| node | IP address of the upstream node. |
+| gateway_group_id | Available only in Enterprise. ID of the gateway group that the HTTP status originates from. |
+| instance_id | Available only in Enterprise. ID of the gateway instance that the HTTP status originates from. |
+| api_product_id | Available only in Enterprise. Product ID that the HTTP status originates from. |
+
+### Labels for `apisix_bandwidth`
+
+The following labels are used to differentiate `apisix_bandwidth` metrics.
+
+| Name | Description |
+| ---------- | ----------------------------------------------------------------------------------------------------------------------------- |
+| type | Type of traffic, `egress` or `ingress`. |
+| route | ID of the route that bandwidth corresponds to when `prefer_name` is `false` (default), and name of the route when `prefer_name` to `true`. Default to an empty string if a request does not match any route. |
+| route_id | Available only in Enterprise. ID of the route that bandwidth corresponds to regardless of the `prefer_name` setting. |
+| service | ID of the service that bandwidth corresponds to when `prefer_name` is `false` (default), and name of the service when `prefer_name` to `true`. Default to the configured value of host on the route if the matched route does not belong to any service. |
+| service_id | Available only in Enterprise. ID of the service that bandwidth corresponds to regardless of the `prefer_name` setting. |
+| consumer | Name of the consumer associated with a request. Default to an empty string if no consumer is associated with the request. |
+| node | IP address of the upstream node. |
+| gateway_group_id | Available only in Enterprise. ID of the gateway group that bandwidth corresponds to. |
+| instance_id | Available only in Enterprise. ID of the gateway instance that bandwidth corresponds to. |
+| api_product_id | Available only in Enterprise. Product ID that bandwidth corresponds to. |
+
+### Labels for `apisix_http_latency`
+
+The following labels are used to differentiate `apisix_http_latency` metrics.
+
+| Name | Description |
+| ---------- | ----------------------------------------------------------------------------------------------------------------------------------- |
+| type | Type of latencies. See [latency types](#latency-types) for details. |
+| route | ID of the route that latencies correspond to when `prefer_name` is `false` (default), and name of the route when `prefer_name` to `true`. Default to an empty string if a request does not match any route. |
+| route_id | Available only in Enterprise. ID of the route that latencies correspond to regardless of the `prefer_name` setting. |
+| service | ID of the service that latencies correspond to when `prefer_name` is `false` (default), and name of the service when `prefer_name` to `true`. Default to the configured value of host on the route if the matched route does not belong to any service. |
+| service_id | Available only in Enterprise. ID of the service that latencies correspond to regardless of the `prefer_name` setting. |
+| consumer | Name of the consumer associated with latencies. Default to an empty string if no consumer is associated with the request. |
+| node | IP address of the upstream node associated with latencies. |
+| gateway_group_id | Available only in Enterprise. ID of the gateway group that latencies correspond to. |
+| instance_id | Available only in Enterprise. ID of the gateway instance that latencies correspond to. |
+| api_product_id | Available only in Enterprise. Product ID that latencies correspond to. |
+
+#### Latency Types
+
+`apisix_http_latency` can be labeled with one of the three types:
+
+* `request` represents the time elapsed between the first byte was read from the client and the log write after the last byte was sent to the client.
+
+* `upstream` represents the time elapsed waiting on responses from the upstream service.
+
+* `apisix` represents the difference between the `request` latency and `upstream` latency.
+
+In other words, the APISIX latency is not only attributed to the Lua processing. It should be understood as follows:
+
+```text
+APISIX latency
+ = downstream request time - upstream response time
+ = downstream traffic latency + NGINX latency
```
-### Specifying `metrics`
-
-For http request related metrics, you could specify extra labels, which match the APISIX variables.
+### Labels for `apisix_upstream_status`
-If you specify label for nonexist APISIX variable, the label value would be "".
+The following labels are used to differentiate `apisix_upstream_status` metrics.
-Currently, only below metrics are supported:
+| Name | Description |
+| ---------- | --------------------------------------------------------------------------------------------------- |
+| name | Resource ID corresponding to the upstream configured with health checks, such as `/apisix/routes/1` and `/apisix/upstreams/1`. |
+| ip | IP address of the upstream node. |
+| port | Port number of the node. |
-* http_status
-* http_latency
-* bandwidth
+## Examples
-Here is a configuration example:
-
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- metrics:
- http_status:
- extra_labels:
- - upstream_addr: $upstream_addr
- - upstream_status: $upstream_status
-```
+The examples below demonstrate how you can work with the `prometheus` plugin for different scenarios.
-### Specifying `default_buckets`
+### Get APISIX Metrics
-`DEFAULT_BUCKETS` is the default value for bucket array in `http_latency` metrics.
+The following example demonstrates how you can get metrics from APISIX.
-You can change the `DEFAULT_BUCKETS` by configuring `default_buckets` attribute in you configuration file.
+The default Prometheus metrics endpoint and other Prometheus related configurations can be found in the [static configuration](#static-configurations). If you would like to customize these configuration, see [configuration files](/apisix/reference/configuration-files#configyaml-and-configyamlexample).
Review Comment:
no related content of static-configurations
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs: improve prometheus plugin docs [apisix]
Posted by "Yilialinn (via GitHub)" <gi...@apache.org>.
Yilialinn commented on code in PR #11874:
URL: https://github.com/apache/apisix/pull/11874#discussion_r1899402049
##########
docs/en/latest/plugins/prometheus.md:
##########
@@ -27,438 +27,423 @@ description: This document contains information about the Apache APISIX promethe
#
-->
-## Description
-
-The `prometheus` Plugin exports metrics in [Prometheus exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/#exposition-formats).
-
-## Attributes
-
-| Name | Type | Required | Default | Description |
-| ----------- | ------- | -------- | ------- | --------------------------------------------------------------------------------- |
-| prefer_name | boolean | False | false | When set to `true`, prints Route/Service name instead of ID in Prometheus metric. |
+<head>
+ <link rel="canonical" href="https://docs.api7.ai/hub/prometheus" />
+</head>
-### Specifying `export_uri`
+## Description
-You can change the default export URI by configuring the `export_uri` attribute under `plugin_attr` in your configuration file (`conf/config.yaml`).
+The `prometheus` plugin provides the capability to integrate APISIX with [Prometheus](https://prometheus.io).
-| Name | Type | Default | Description |
-| ---------- | ------ | ---------------------------- | ------------------------------------- |
-| export_uri | string | "/apisix/prometheus/metrics" | URI to export the Prometheus metrics. |
+After enabling the plugin, APISIX will start collecting relevant metrics, such as API requests and latencies, and exporting them in a [text-based exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/#exposition-formats) to Prometheus. You can then create event monitoring and alerting in Prometheus to monitor the health of your API gateway and APIs.
-Here is a configuration example:
+## Attributes
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- export_uri: /apisix/metrics
+There are different types of metrics in Prometheus. To understand their differences, see [metrics types](https://prometheus.io/docs/concepts/metric_types/).
+
+The following metrics are exported by the `prometheus` plugin by default. See [get APISIX metrics](#get-apisix-metrics) for an example. Note that some metrics, such as `apisix_batch_process_entries`, are not readily visible if there are no data.
+
+
+| Name | Type | Description |
+| ------------------------------ | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| apisix_bandwidth | counter | Total amount of traffic flowing through APISIX in bytes. |
+| apisix_etcd_modify_indexes | gauge | Number of changes to etcd by APISIX keys. |
+| apisix_batch_process_entries | gauge | Number of remaining entries in a batch when sending data in batches, such as with `http logger`, and other logging plugins. |
+| apisix_etcd_reachable | gauge | Whether APISIX can reach etcd. A value of `1` represents reachable and `0` represents unreachable. |
+| apisix_http_status | counter | HTTP status codes returned from upstream services. |
+| apisix_http_requests_total | gauge | Number of HTTP requests from clients. |
+| apisix_nginx_http_current_connections | gauge | Number of current connections with clients. |
+| apisix_nginx_metric_errors_total | counter | Total number of `nginx-lua-prometheus` errors. |
+| apisix_http_latency | histogram | HTTP request latency in milliseconds. |
+| apisix_node_info | gauge | Information of the APISIX node, such as host name. |
+| apisix_shared_dict_capacity_bytes | gauge | The total capacity of an [NGINX shared dictionary](https://github.com/openresty/lua-nginx-module#ngxshareddict). |
+| apisix_shared_dict_free_space_bytes | gauge | The remaining space in an [NGINX shared dictionary](https://github.com/openresty/lua-nginx-module#ngxshareddict). |
+| apisix_upstream_status | gauge | Health check status of upstream nodes, available if health checks are configured on the upstream. A value of `1` represents healthy and `0` represents unhealthy. |
+| apisix_stream_connection_total | counter | Total number of connections handled per stream route. |
+
+## Labels
+
+[Labels](https://prometheus.io/docs/practices/naming/#labels) are attributes of metrics that are used to differentiate metrics.
+
+For example, the `apisix_http_status` metric can be labeled with `route` information to identify which route the HTTP status originates from.
+
+The following are labels for a non-exhaustive list of APISIX metrics and their descriptions.
+
+### Labels for `apisix_http_status`
+
+The following labels are used to differentiate `apisix_http_status` metrics.
+
+| Name | Description |
+| ------------ | ----------------------------------------------------------------------------------------------------------------------------- |
+| code | HTTP response code returned by the upstream node. |
+| route | ID of the route that the HTTP status originates from when `prefer_name` is `false` (default), and name of the route when `prefer_name` to `true`. Default to an empty string if a request does not match any route. |
+| route_id | Available only in Enterprise. ID of the route that the HTTP status originates from regardless of the `prefer_name` setting. |
+| matched_uri | URI of the route that matches the request. Default to an empty string if a request does not match any route. |
+| matched_host | Host of the route that matches the request. Default to an empty string if a request does not match any route, or host is not configured on the route. |
+| service | ID of the service that the HTTP status originates from when `prefer_name` is `false` (default), and name of the service when `prefer_name` to `true`. Default to the configured value of host on the route if the matched route does not belong to any service. |
+| service_id | Available only in Enterprise. ID of the service that the HTTP status originates from regardless of the `prefer_name` setting. |
+| consumer | Name of the consumer associated with a request. Default to an empty string if no consumer is associated with the request. |
+| node | IP address of the upstream node. |
+| gateway_group_id | Available only in Enterprise. ID of the gateway group that the HTTP status originates from. |
+| instance_id | Available only in Enterprise. ID of the gateway instance that the HTTP status originates from. |
+| api_product_id | Available only in Enterprise. Product ID that the HTTP status originates from. |
+
+### Labels for `apisix_bandwidth`
+
+The following labels are used to differentiate `apisix_bandwidth` metrics.
+
+| Name | Description |
+| ---------- | ----------------------------------------------------------------------------------------------------------------------------- |
+| type | Type of traffic, `egress` or `ingress`. |
+| route | ID of the route that bandwidth corresponds to when `prefer_name` is `false` (default), and name of the route when `prefer_name` to `true`. Default to an empty string if a request does not match any route. |
+| route_id | Available only in Enterprise. ID of the route that bandwidth corresponds to regardless of the `prefer_name` setting. |
+| service | ID of the service that bandwidth corresponds to when `prefer_name` is `false` (default), and name of the service when `prefer_name` to `true`. Default to the configured value of host on the route if the matched route does not belong to any service. |
+| service_id | Available only in Enterprise. ID of the service that bandwidth corresponds to regardless of the `prefer_name` setting. |
+| consumer | Name of the consumer associated with a request. Default to an empty string if no consumer is associated with the request. |
+| node | IP address of the upstream node. |
+| gateway_group_id | Available only in Enterprise. ID of the gateway group that bandwidth corresponds to. |
+| instance_id | Available only in Enterprise. ID of the gateway instance that bandwidth corresponds to. |
+| api_product_id | Available only in Enterprise. Product ID that bandwidth corresponds to. |
+
+### Labels for `apisix_http_latency`
+
+The following labels are used to differentiate `apisix_http_latency` metrics.
+
+| Name | Description |
+| ---------- | ----------------------------------------------------------------------------------------------------------------------------------- |
+| type | Type of latencies. See [latency types](#latency-types) for details. |
+| route | ID of the route that latencies correspond to when `prefer_name` is `false` (default), and name of the route when `prefer_name` to `true`. Default to an empty string if a request does not match any route. |
+| route_id | Available only in Enterprise. ID of the route that latencies correspond to regardless of the `prefer_name` setting. |
+| service | ID of the service that latencies correspond to when `prefer_name` is `false` (default), and name of the service when `prefer_name` to `true`. Default to the configured value of host on the route if the matched route does not belong to any service. |
+| service_id | Available only in Enterprise. ID of the service that latencies correspond to regardless of the `prefer_name` setting. |
+| consumer | Name of the consumer associated with latencies. Default to an empty string if no consumer is associated with the request. |
+| node | IP address of the upstream node associated with latencies. |
+| gateway_group_id | Available only in Enterprise. ID of the gateway group that latencies correspond to. |
+| instance_id | Available only in Enterprise. ID of the gateway instance that latencies correspond to. |
+| api_product_id | Available only in Enterprise. Product ID that latencies correspond to. |
+
+#### Latency Types
+
+`apisix_http_latency` can be labeled with one of the three types:
+
+* `request` represents the time elapsed between the first byte was read from the client and the log write after the last byte was sent to the client.
+
+* `upstream` represents the time elapsed waiting on responses from the upstream service.
+
+* `apisix` represents the difference between the `request` latency and `upstream` latency.
+
+In other words, the APISIX latency is not only attributed to the Lua processing. It should be understood as follows:
+
+```text
+APISIX latency
+ = downstream request time - upstream response time
+ = downstream traffic latency + NGINX latency
```
-### Specifying `metrics`
-
-For http request related metrics, you could specify extra labels, which match the APISIX variables.
+### Labels for `apisix_upstream_status`
-If you specify label for nonexist APISIX variable, the label value would be "".
+The following labels are used to differentiate `apisix_upstream_status` metrics.
-Currently, only below metrics are supported:
+| Name | Description |
+| ---------- | --------------------------------------------------------------------------------------------------- |
+| name | Resource ID corresponding to the upstream configured with health checks, such as `/apisix/routes/1` and `/apisix/upstreams/1`. |
+| ip | IP address of the upstream node. |
+| port | Port number of the node. |
-* http_status
-* http_latency
-* bandwidth
+## Examples
-Here is a configuration example:
-
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- metrics:
- http_status:
- extra_labels:
- - upstream_addr: $upstream_addr
- - upstream_status: $upstream_status
-```
+The examples below demonstrate how you can work with the `prometheus` plugin for different scenarios.
-### Specifying `default_buckets`
+### Get APISIX Metrics
-`DEFAULT_BUCKETS` is the default value for bucket array in `http_latency` metrics.
+The following example demonstrates how you can get metrics from APISIX.
-You can change the `DEFAULT_BUCKETS` by configuring `default_buckets` attribute in you configuration file.
+The default Prometheus metrics endpoint and other Prometheus related configurations can be found in the [static configuration](#static-configurations). If you would like to customize these configuration, see [configuration files](/apisix/reference/configuration-files#configyaml-and-configyamlexample).
Review Comment:
no related content
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] docs: improve prometheus plugin docs [apisix]
Posted by "kayx23 (via GitHub)" <gi...@apache.org>.
kayx23 commented on code in PR #11874:
URL: https://github.com/apache/apisix/pull/11874#discussion_r1900606576
##########
docs/zh/latest/plugins/prometheus.md:
##########
@@ -27,408 +27,448 @@ description: 本文将介绍 API 网关 Apache APISIX 如何通过 prometheus
#
-->
+<head>
+ <link rel="canonical" href="https://docs.api7.ai/hub/prometheus" />
+</head>
+
## 描述
-`prometheus` 插件以 [Prometheus 文档](https://prometheus.io/docs/instrumenting/exposition_formats/#exposition-formats)规定的格式上报指标到 Prometheus 中。
+`prometheus` 插件提供将 APISIX 与 Prometheus 集成的能力。
+
+启用该插件后，APISIX 将开始收集相关指标，例如 API 请求和延迟，并以[基于文本的展示格式](https://prometheus.io/docs/instrumenting/exposition_formats/#exposition-formats)导出到 Prometheus。然后，您可以在 Prometheus 中创建事件监控和警报，以监控 API 网关和 API 的健康状况。
+
+## 静态配置
+
+默认情况下，已在默认配置文件 [`config.lua`](https://github.com/apache/apisix/blob/master/apisix/cli/config.lua) 中对 `prometheus` 进行预配置。
+
+要自定义这些值，请将相应的配置添加到 config.yaml 中。例如：
+
+```yaml
+plugin_attr:
+ prometheus: # 插件: prometheus 属性
+ export_uri: /apisix/prometheus/metrics # 设置 Prometheus 指标端点的 URI。
+ metric_prefix: apisix_ # 设置 APISIX 生成的 Prometheus 指标的前缀。
+ enable_export_server: true # 启用 Prometheus 导出服务器。
+ export_addr: # 设置 Prometheus 导出服务器的地址。
+ ip: 127.0.0.1 # 设置 IP。
+ port: 9091 # 设置端口。
+ # metrics: # 为指标创建额外的标签。
+ # http_status: # 这些指标将以 `apisix_` 为前缀。
+ # extra_labels: # 设置 http_status 指标的额外标签。
+ # - upstream_addr: $upstream_addr
+ # - status: $upstream_status
+ # expire: 0 # 指标的过期时间（秒）。
+ # 0 表示指标不会过期。
+ # http_latency:
+ # extra_labels: # 设置 http_latency 指标的额外标签。
+ # - upstream_addr: $upstream_addr
+ # expire: 0 # 指标的过期时间（秒）。
+ # 0 表示指标不会过期。
+ # bandwidth:
+ # extra_labels: # 设置 bandwidth 指标的额外标签。
+ # - upstream_addr: $upstream_addr
+ # expire: 0 # 指标的过期时间（秒）。
+ # 0 表示指标不会过期。
+ # default_buckets: # 设置 `http_latency` 指标直方图的默认桶。
+ # - 10
+ # - 50
+ # - 100
+ # - 200
+ # - 500
+ # - 1000
+ # - 2000
+ # - 5000
+ # - 10000
+ # - 30000
+ # - 60000
+ # - 500
+```
+
+您可以使用 [Nginx 变量](https://nginx.org/en/docs/http/ngx_http_core_module.html)创建 `extra_labels`。请参见[添加额外标签](#add-extra-labels-for-metrics)。
+
+重新加载 APISIX 以使更改生效。
## 属性
| 名称 | 类型 | 必选项 | 默认值 | 描述 |
| ------------ | --------| ------ | ------ | ----------------------------------------------------- |
-| prefer_name | boolean | 否 | false | 当设置为 `true` 时，将使用路由或服务的 `name` 标识请求所命中的路由或服务，否则使用其 `id`。 |
+|`prefer_name` | boolean | 否 | False | 当设置为 `true` 时，则在`prometheus` 指标中导出路由/服务名称而非它们的 `id`。 |
-:::note
+## 指标
-多个路由或服务可以设置为相同的名称，所以当设置 `prefer_name` 为 `true` 时，请规范路由和服务的命名，否则容易引起误解。
+Prometheus 中有不同类型的指标。要了解它们之间的区别，请参见[指标类型](https://prometheus.io/docs/concepts/metric_types/)。
-:::
+以下是 `prometheus` 插件默认导出的指标。有关示例，请参见[获取 APISIX 指标](#get-apisix-metrics)。请注意，一些指标，例如 `apisix_batch_process_entries`，如果没有数据，将不可见。
-### 如何修改暴露指标的 `export_uri`
+| 名称 | 类型 | 描述 |
+| ----------------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| apisix_bandwidth | counter | APISIX 中每个服务消耗的总流量（字节）。 |
+| apisix_etcd_modify_indexes | gauge | APISIX 键的 etcd 修改次数。 |
+| apisix_batch_process_entries | gauge | 发送数据时批处理中的剩余条目数，例如使用 `http logger` 和其他日志插件。 |
+| apisix_etcd_reachable | gauge | APISIX 是否可以访问 etcd。值为 `1` 表示可达，`0` 表示不可达。 |
+| apisix_http_status | counter | 从上游服务返回的 HTTP 状态代码。 |
+| apisix_http_requests_total | gauge | 来自客户端的 HTTP 请求数量。 |
+| apisix_nginx_http_current_connections | gauge | 当前与客户端的连接数量。 |
+| apisix_nginx_metric_errors_total | counter | `nginx-lua-prometheus` 错误的总数。 |
+| apisix_http_latency | histogram | HTTP 请求延迟（毫秒）。 |
+| apisix_node_info | gauge | APISIX 节点的信息，例如主机名。 |
+| apisix_shared_dict_capacity_bytes | gauge | [NGINX 共享字典](https://github.com/openresty/lua-nginx-module#ngxshareddict) 的总容量。 |
+| apisix_shared_dict_free_space_bytes | gauge | [NGINX 共享字典](https://github.com/openresty/lua-nginx-module#ngxshareddict) 中剩余的空间。 |
+| apisix_upstream_status | gauge | 上游节点的健康检查状态，如果在上游配置了健康检查，则可用。值为 `1` 表示健康，`0` 表示不健康。 |
+| apisix_stream_connection_total | counter | 每个流路由处理的总连接数。 |
-你可以在配置文件 `./conf/config.yaml` 的 `plugin_attr` 列表下修改默认的 URI。
+## 标签
-| 名称 | 类型 | 默认值 | 描述 |
-| ---------- | ------ | ---------------------------- | --------------------------- |
-| export_uri | string | "/apisix/prometheus/metrics" | 暴露 Prometheus 指标的 URI。 |
+[标签](https://prometheus.io/docs/practices/naming/#labels) 是指标的属性，用于区分指标。
-配置示例如下：
+例如，`apisix_http_status` 指标可以使用 `route` 信息进行标记，以识别 HTTP 状态的来源路由。
-```yaml title="./conf/config.yaml"
-plugin_attr:
- prometheus:
- export_uri: /apisix/metrics
-```
+以下是 APISIX 指标的非详尽标签及其描述。
-### 如何修改延迟指标中的 `default_buckets`
+### `apisix_http_status` 的标签
-`DEFAULT_BUCKETS` 是 `http_latency` 指标中 bucket 数组的默认值。
+以下标签用于区分 `apisix_http_status` 指标。
-你可以通过修改配置文件中的 `default_buckets` 来重新指定 `DEFAULT_BUCKETS`
+| 名称 | 描述 |
+| ------ | ---------------------------------------------------------------------------------------------------------------------- |
+| code | 上游节点返回的 HTTP 响应代码。 |
+| route | HTTP 状态来源的路由 ID，当 `prefer_name` 为 `false`（默认）时，使用路由 ID，当 `prefer_name` 为 `true` 时，使用路由名称。如果请求不匹配任何路由，则默认为空字符串。 |
+| matched_uri | 匹配请求的路由 URI。如果请求不匹配任何路由，则默认为空字符串。 |
+| matched_host | 匹配请求的路由主机。如果请求不匹配任何路由，或路由未配置主机，则默认为空字符串。 |
+| service | HTTP 状态来源的服务 ID，当 `prefer_name` 为 `false`（默认）时，使用服务 ID，当 `prefer_name` 为 `true` 时，使用服务名称。如果匹配的路由不属于任何服务，则默认为路由上配置的主机值。 |
+| consumer | 与请求关联的消费者名称。如果请求没有与之关联的消费者，则默认为空字符串。 |
+| node | 上游节点的 IP 地址。 |
-配置示例如下：
+### `apisix_bandwidth` 的标签
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- default_buckets:
- - 15
- - 55
- - 105
- - 205
- - 505
-```
+以下标签用于区分 `apisix_bandwidth` 指标。
-### 如何修改指标的 `expire`
+| 名称 | 描述 |
+| ------ | ---------------------------------------------------------------------------------------------------------------------- |
+| type | 流量类型，`egress` 或 `ingress`。 |
+| route | 带宽对应的路由 ID，当 `prefer_name` 为 `false`（默认）时，使用路由 ID，当 `prefer_name` 为 `true` 时，使用路由名称。如果请求不匹配任何路由，则默认为空字符串。 |
+| service | 带宽对应的服务 ID，当 `prefer_name` 为 `false`（默认）时，使用服务 ID，当 `prefer_name` 为 `true` 时，使用服务名称。如果匹配的路由不属于任何服务，则默认为路由上配置的主机值。 |
+| consumer | 与请求关联的消费者名称。如果请求没有与之关联的消费者，则默认为空字符串。 |
+| node | 上游节点的 IP 地址。 |
-`expire` 用于设置 `apisix_http_status`、`apisix_bandwidth` 和 `apisix_http_latency` 指标的过期时间（以秒为单位）。当设置为 0 时，指标不会过期。
+### `apisix_http_latency` 的标签
-配置示例如下：
+以下标签用于区分 `apisix_http_latency` 指标。
-```yaml title="conf/config.yaml"
-plugin_attr:
- prometheus:
- expire: 86400
-```
+| 名称 | 描述 |
+| ------ | ---------------------------------------------------------------------------------------------------------------------- |
+| type | 延迟类型。有关详细信息，请参见 [延迟类型](#latency-types)。 |
Review Comment:
```suggestion
| type | 延迟类型。有关详细信息，请参见 [延迟类型](#延迟类型)。 |
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org