You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/04/03 03:21:30 UTC
[GitHub] [pulsar] Anonymitaet opened a new pull request #10133: [Doc] Add monitoring guidelines for function and connector
Anonymitaet opened a new pull request #10133:
URL: https://github.com/apache/pulsar/pull/10133
Fix https://github.com/apache/pulsar/issues/9932
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on pull request #10133: [Doc] Add monitoring guidelines for function and connector
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on pull request #10133:
URL: https://github.com/apache/pulsar/pull/10133#issuecomment-813754058
@Huanli-Meng thanks for your comments, I've incorporated them, PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] freeznet commented on a change in pull request #10133: [Doc] Add monitoring guidelines for function and connector
Posted by GitBox <gi...@apache.org>.
freeznet commented on a change in pull request #10133:
URL: https://github.com/apache/pulsar/pull/10133#discussion_r606614822
##########
File path: site2/docs/deploy-monitoring.md
##########
@@ -69,6 +69,26 @@ brk_ml_cursor_persistZookeeperErrors(namespace="", ledger_name="", cursor_name:"
Those metrics are added in the Prometheus interface, you can monitor and check the metrics stats in the Grafana.
+### Function & connector stats
+
+You can collect functions worker stats from `functions-worker` and export the metrics in JSON formats, which contain functions worker JVM metrics.
+
+```
+pulsar-admin functions-worker monitoring-metrics
+```
+
+You can collect functions and connectors metrics from `functions-worker` and export the metrics in JSON formats.
+
+```
+pulsar-admin functions-worker function-stats
+```
+
+The aggregated functions and connectors metrics can be exposed in Prometheus formats as below. You can get `FUNCTIONS_WORKER_ADDRESS` and `WORKER_PORT` from the `functions_worker.yml` file.
Review comment:
`FUNCTIONS_WORKER_ADDRESS` is not in `functions_worker.yml`, it is related how user config their functions worker, related to [http://pulsar.apache.org/docs/en/next/functions-worker/](http://pulsar.apache.org/docs/en/next/functions-worker/).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on a change in pull request #10133: [Doc] Add monitoring guidelines for function and connector
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on a change in pull request #10133:
URL: https://github.com/apache/pulsar/pull/10133#discussion_r608459661
##########
File path: site2/docs/deploy-monitoring.md
##########
@@ -69,6 +69,26 @@ brk_ml_cursor_persistZookeeperErrors(namespace="", ledger_name="", cursor_name:"
Those metrics are added in the Prometheus interface, you can monitor and check the metrics stats in the Grafana.
+### Function & connector stats
+
+You can collect functions worker stats from `functions-worker` and export the metrics in JSON formats, which contain functions worker JVM metrics.
+
+```
+pulsar-admin functions-worker monitoring-metrics
+```
+
+You can collect functions and connectors metrics from `functions-worker` and export the metrics in JSON formats.
+
+```
+pulsar-admin functions-worker function-stats
+```
+
+The aggregated functions and connectors metrics can be exposed in Prometheus formats as below. You can get `FUNCTIONS_WORKER_ADDRESS` and `WORKER_PORT` from the `functions_worker.yml` file.
Review comment:
@freeznet OK, I've added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] freeznet commented on a change in pull request #10133: [Doc] Add monitoring guidelines for function and connector
Posted by GitBox <gi...@apache.org>.
freeznet commented on a change in pull request #10133:
URL: https://github.com/apache/pulsar/pull/10133#discussion_r608452767
##########
File path: site2/docs/deploy-monitoring.md
##########
@@ -69,6 +69,26 @@ brk_ml_cursor_persistZookeeperErrors(namespace="", ledger_name="", cursor_name:"
Those metrics are added in the Prometheus interface, you can monitor and check the metrics stats in the Grafana.
+### Function & connector stats
+
+You can collect functions worker stats from `functions-worker` and export the metrics in JSON formats, which contain functions worker JVM metrics.
+
+```
+pulsar-admin functions-worker monitoring-metrics
+```
+
+You can collect functions and connectors metrics from `functions-worker` and export the metrics in JSON formats.
+
+```
+pulsar-admin functions-worker function-stats
+```
+
+The aggregated functions and connectors metrics can be exposed in Prometheus formats as below. You can get `FUNCTIONS_WORKER_ADDRESS` and `WORKER_PORT` from the `functions_worker.yml` file.
Review comment:
I suggest to add a link to functions-worker doc, should enough to let reader understand.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Huanli-Meng commented on a change in pull request #10133: [Doc] Add monitoring guidelines for function and connector
Posted by GitBox <gi...@apache.org>.
Huanli-Meng commented on a change in pull request #10133:
URL: https://github.com/apache/pulsar/pull/10133#discussion_r607403218
##########
File path: site2/docs/reference-metrics.md
##########
@@ -416,6 +416,50 @@ All the Pulsar Functions metrics are labelled with the following labels:
| pulsar_function_last_invocation | Gauge | The timestamp of the last invocation of the function. |
| pulsar_function_received_total | Counter | Total number of messages received from source. |
| pulsar_function_received_total_1min | Counter | Total number of messages received from source in the last 1 minute. |
+pulsar_function_user_metric_ | Summary|User defined metric.
+
+## Connectors
+
+All the Pulsar connector metrics are labelled with the following labels:
+
+- *cluster*: `cluster=${pulsar_cluster}`. `${pulsar_cluster}` is the cluster name that you have configured in the `broker.conf` file.
+- *namespace*: `namespace=${pulsar_namespace}`. `${pulsar_namespace}` is the namespace name.
+
+Connector metrics contain **source** metrics and **sink** metrics.
+
+- **Source** metrics
+
+ | Name | Type | Description |
+ |---|---|---|
+ pulsar_source_written_total|Counter|Total number of records written to a Pulsar topic.
Review comment:
I think it's better to use "the total number of...", same comments for the following similar descriptions.
##########
File path: site2/docs/reference-metrics.md
##########
@@ -416,6 +416,50 @@ All the Pulsar Functions metrics are labelled with the following labels:
| pulsar_function_last_invocation | Gauge | The timestamp of the last invocation of the function. |
| pulsar_function_received_total | Counter | Total number of messages received from source. |
| pulsar_function_received_total_1min | Counter | Total number of messages received from source in the last 1 minute. |
+pulsar_function_user_metric_ | Summary|User defined metric.
+
+## Connectors
+
+All the Pulsar connector metrics are labelled with the following labels:
+
+- *cluster*: `cluster=${pulsar_cluster}`. `${pulsar_cluster}` is the cluster name that you have configured in the `broker.conf` file.
+- *namespace*: `namespace=${pulsar_namespace}`. `${pulsar_namespace}` is the namespace name.
+
+Connector metrics contain **source** metrics and **sink** metrics.
+
+- **Source** metrics
+
+ | Name | Type | Description |
+ |---|---|---|
+ pulsar_source_written_total|Counter|Total number of records written to a Pulsar topic.
+ pulsar_source_written_total_1min|Counter|Total number of records written to a Pulsar topic in the last 1 minute.
+ pulsar_source_received_total|Counter|Total number of records received from source.
+ pulsar_source_received_total_1min|Counter|Total number of records received from source in the last 1 minute.
+ pulsar_source_last_invocation|Gauge|The timestamp of the last invocation of the source.
+ pulsar_source_source_exception|Gauge|Exception from a source.
+ pulsar_source_source_exceptions_total|Counter|Total number of source exceptions.
+ pulsar_source_source_exceptions_total_1min |Counter|Total number of source exceptions in the last 1 minute.
+ pulsar_source_system_exception|Gauge|Exception from system code.
+ pulsar_source_system_exceptions_total|Counter|Total number of system exceptions.
+ pulsar_source_system_exceptions_total_1min|Counter|Total number of system exceptions in the last 1 minute.
+ pulsar_source_user_metric_ | Summary|User defined metric.
Review comment:
```suggestion
pulsar_source_user_metric_ | Summary|User-defined metrics.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on a change in pull request #10133: [Doc] Add monitoring guidelines for function and connector
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on a change in pull request #10133:
URL: https://github.com/apache/pulsar/pull/10133#discussion_r606649997
##########
File path: site2/docs/deploy-monitoring.md
##########
@@ -69,6 +69,26 @@ brk_ml_cursor_persistZookeeperErrors(namespace="", ledger_name="", cursor_name:"
Those metrics are added in the Prometheus interface, you can monitor and check the metrics stats in the Grafana.
+### Function & connector stats
+
+You can collect functions worker stats from `functions-worker` and export the metrics in JSON formats, which contain functions worker JVM metrics.
+
+```
+pulsar-admin functions-worker monitoring-metrics
+```
+
+You can collect functions and connectors metrics from `functions-worker` and export the metrics in JSON formats.
+
+```
+pulsar-admin functions-worker function-stats
+```
+
+The aggregated functions and connectors metrics can be exposed in Prometheus formats as below. You can get `FUNCTIONS_WORKER_ADDRESS` and `WORKER_PORT` from the `functions_worker.yml` file.
Review comment:
@freeznet thanks for the explanations. Should we need to tell readers how to get `FUNCTIONS_WORKER_ADDRESS` here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on a change in pull request #10133: [Doc] Add monitoring guidelines for function and connector
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on a change in pull request #10133:
URL: https://github.com/apache/pulsar/pull/10133#discussion_r606609194
##########
File path: site2/docs/deploy-monitoring.md
##########
@@ -69,6 +69,26 @@ brk_ml_cursor_persistZookeeperErrors(namespace="", ledger_name="", cursor_name:"
Those metrics are added in the Prometheus interface, you can monitor and check the metrics stats in the Grafana.
+### Function & connector stats
+
+You can collect functions worker stats from `functions-worker` and export the metrics in JSON formats, which contain functions worker JVM metrics.
+
+```
+pulsar-admin functions-worker monitoring-metrics
+```
+
+You can collect functions and connectors metrics from `functions-worker` and export the metrics in JSON formats.
+
+```
+pulsar-admin functions-worker function-stats
+```
+
+The aggregated functions and connectors metrics can be exposed in Prometheus formats as below. You can get `FUNCTIONS_WORKER_ADDRESS` and `WORKER_PORT` from the `functions_worker.yml` file.
Review comment:
@freeznet I add `FUNCTIONS_WORKER_ADDRESS` here, is it correct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet merged pull request #10133: [Doc] Add monitoring guidelines for function and connector
Posted by GitBox <gi...@apache.org>.
Anonymitaet merged pull request #10133:
URL: https://github.com/apache/pulsar/pull/10133
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on a change in pull request #10133: [Doc] Add monitoring guidelines for function and connector
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on a change in pull request #10133:
URL: https://github.com/apache/pulsar/pull/10133#discussion_r607429832
##########
File path: site2/docs/deploy-monitoring.md
##########
@@ -69,6 +69,26 @@ brk_ml_cursor_persistZookeeperErrors(namespace="", ledger_name="", cursor_name:"
Those metrics are added in the Prometheus interface, you can monitor and check the metrics stats in the Grafana.
+### Function & connector stats
+
+You can collect functions worker stats from `functions-worker` and export the metrics in JSON formats, which contain functions worker JVM metrics.
+
+```
+pulsar-admin functions-worker monitoring-metrics
+```
+
+You can collect functions and connectors metrics from `functions-worker` and export the metrics in JSON formats.
+
+```
+pulsar-admin functions-worker function-stats
+```
+
+The aggregated functions and connectors metrics can be exposed in Prometheus formats as below. You can get `FUNCTIONS_WORKER_ADDRESS` and `WORKER_PORT` from the `functions_worker.yml` file.
Review comment:
@freeznet any feedback?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on pull request #10133: [Doc] Add monitoring guidelines for function and connector
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on pull request #10133:
URL: https://github.com/apache/pulsar/pull/10133#issuecomment-812802562
@freeznet @Huanli-Meng @Jennifer88huang could you please help review? Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org