You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/03/26 08:19:12 UTC
[GitHub] [pulsar] hangc0276 opened a new pull request #6615: Export
ManagedLedgerCache metric to prometheus
hangc0276 opened a new pull request #6615: Export ManagedLedgerCache metric to prometheus
URL: https://github.com/apache/pulsar/pull/6615
### Motivation
The managed ledger read cache monitor metric is export via `/admin/broker-stats/metrics` with json format, it is hard to parse, collect and display, what's more the read cache is a very import module for message consuming throughput and latency. So collect and display the read cache metrics is extremely urgent for pulsar in production.
### Changes
I parse the json format metric to prometheus message type and export to prometheus monitor port, so those metrics can be displayed in grafana.
Some of the metric name contains dot, but promethues metric name rule only accept `[a-zA-Z0-9_:]`, so i replace `dot` to `:` in metric name to avoid prometheus parse error
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] jiazhai commented on issue #6615: Export
ManagedLedgerCache metric to prometheus
Posted by GitBox <gi...@apache.org>.
jiazhai commented on issue #6615: Export ManagedLedgerCache metric to prometheus
URL: https://github.com/apache/pulsar/pull/6615#issuecomment-605416779
/pulsarbot run-failure-checks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] hangc0276 closed pull request #6615: Export
ManagedLedgerCache metric to prometheus
Posted by GitBox <gi...@apache.org>.
hangc0276 closed pull request #6615: Export ManagedLedgerCache metric to prometheus
URL: https://github.com/apache/pulsar/pull/6615
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] codelipenghui commented on a change in pull request #6615:
Export ManagedLedgerCache metric to prometheus
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on a change in pull request #6615: Export ManagedLedgerCache metric to prometheus
URL: https://github.com/apache/pulsar/pull/6615#discussion_r399739354
##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricsGenerator.java
##########
@@ -75,12 +81,54 @@ public static void generate(PulsarService pulsar, boolean includeTopicMetrics, b
FunctionsStatsGenerator.generate(pulsar.getWorkerService(),
pulsar.getConfiguration().getClusterName(), stream);
+ generateBrokerBasicMetrics(pulsar, stream);
+
out.write(buf.array(), buf.arrayOffset(), buf.readableBytes());
} finally {
buf.release();
}
}
+ private static void generateBrokerBasicMetrics(PulsarService pulsar, SimpleTextOutputStream stream) {
+ String clusterName = pulsar.getConfiguration().getClusterName();
+ // generate managedLedgerCache metrics
+ parseMetricsToPrometheusMetrics(new ManagedLedgerCacheMetrics(pulsar).generate(),
+ clusterName, "managedLedgerCache", Collector.Type.GAUGE, stream);
+
+ // generate managerLedger metrics
+ parseMetricsToPrometheusMetrics(new ManagedLedgerMetrics(pulsar).generate(),
+ clusterName, "manageLedger", Collector.Type.GAUGE, stream);
+
+ // generate brokerService topic metrics
+ parseMetricsToPrometheusMetrics(pulsar.getBrokerService().getTopicMetrics(),
+ clusterName, "brokerService", Collector.Type.GAUGE, stream);
+
+ // generate load manager loadBalancing metrics
+ parseMetricsToPrometheusMetrics(pulsar.getLoadManager().get().getLoadBalancingMetrics(),
+ clusterName, "loadBalancing", Collector.Type.GAUGE, stream);
+ }
+
+ private static void parseMetricsToPrometheusMetrics(Collection<Metrics> metrics, String cluster, String metricName,
+ Collector.Type metricType, SimpleTextOutputStream stream) {
+ stream.write("# TYPE ").write(metricName).write(' ').write(getTypeStr(metricType)).write('\n');
+ for (Metrics metrics1 : metrics) {
+ String labels = "{cluster=\"" + cluster + "\"";
Review comment:
Can it be replaced by stream.write?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] codelipenghui commented on a change in pull request #6615:
Export ManagedLedgerCache metric to prometheus
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on a change in pull request #6615: Export ManagedLedgerCache metric to prometheus
URL: https://github.com/apache/pulsar/pull/6615#discussion_r399740905
##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricsGenerator.java
##########
@@ -75,12 +81,54 @@ public static void generate(PulsarService pulsar, boolean includeTopicMetrics, b
FunctionsStatsGenerator.generate(pulsar.getWorkerService(),
pulsar.getConfiguration().getClusterName(), stream);
+ generateBrokerBasicMetrics(pulsar, stream);
+
out.write(buf.array(), buf.arrayOffset(), buf.readableBytes());
} finally {
buf.release();
}
}
+ private static void generateBrokerBasicMetrics(PulsarService pulsar, SimpleTextOutputStream stream) {
+ String clusterName = pulsar.getConfiguration().getClusterName();
+ // generate managedLedgerCache metrics
+ parseMetricsToPrometheusMetrics(new ManagedLedgerCacheMetrics(pulsar).generate(),
+ clusterName, "managedLedgerCache", Collector.Type.GAUGE, stream);
+
+ // generate managerLedger metrics
+ parseMetricsToPrometheusMetrics(new ManagedLedgerMetrics(pulsar).generate(),
+ clusterName, "manageLedger", Collector.Type.GAUGE, stream);
+
+ // generate brokerService topic metrics
+ parseMetricsToPrometheusMetrics(pulsar.getBrokerService().getTopicMetrics(),
+ clusterName, "brokerService", Collector.Type.GAUGE, stream);
+
+ // generate load manager loadBalancing metrics
+ parseMetricsToPrometheusMetrics(pulsar.getLoadManager().get().getLoadBalancingMetrics(),
+ clusterName, "loadBalancing", Collector.Type.GAUGE, stream);
+ }
+
+ private static void parseMetricsToPrometheusMetrics(Collection<Metrics> metrics, String cluster, String metricName,
+ Collector.Type metricType, SimpleTextOutputStream stream) {
+ stream.write("# TYPE ").write(metricName).write(' ').write(getTypeStr(metricType)).write('\n');
+ for (Metrics metrics1 : metrics) {
+ String labels = "{cluster=\"" + cluster + "\"";
+ for (Map.Entry<String, String> entry : metrics1.getDimensions().entrySet()) {
+ if (entry.getKey().isEmpty() || "cluster".equals(entry.getKey())) {
+ continue;
+ }
+ labels += ", " + entry.getKey() + "=\"" + entry.getValue() + "\"";
+ }
+ labels += "} ";
+
+ for (Map.Entry<String, Object> entry : metrics1.getMetrics().entrySet()) {
+ stream.write(entry.getKey().replace(".", ":")).write(labels)
Review comment:
If we use metrics key in `ManagedLedgerMetrics` or `ManagedLedgerCacheMetrics` to convert to OpenMetrics(Prometheus) data format metrics directly, this may cause inconsistencies with the metrics format that has been exposed in Prometheus. For example, we use "pulsar_" as the prefix of all broker metrics. If we expose some broker level metrics with `brk_` prefix, this can be confusing for users.
So, I think we need to be as consistent as possible with the current format.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] codelipenghui commented on a change in pull request #6615:
Export ManagedLedgerCache metric to prometheus
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on a change in pull request #6615: Export ManagedLedgerCache metric to prometheus
URL: https://github.com/apache/pulsar/pull/6615#discussion_r399741095
##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricsGenerator.java
##########
@@ -75,12 +81,54 @@ public static void generate(PulsarService pulsar, boolean includeTopicMetrics, b
FunctionsStatsGenerator.generate(pulsar.getWorkerService(),
pulsar.getConfiguration().getClusterName(), stream);
+ generateBrokerBasicMetrics(pulsar, stream);
+
out.write(buf.array(), buf.arrayOffset(), buf.readableBytes());
} finally {
buf.release();
}
}
+ private static void generateBrokerBasicMetrics(PulsarService pulsar, SimpleTextOutputStream stream) {
+ String clusterName = pulsar.getConfiguration().getClusterName();
+ // generate managedLedgerCache metrics
+ parseMetricsToPrometheusMetrics(new ManagedLedgerCacheMetrics(pulsar).generate(),
+ clusterName, "managedLedgerCache", Collector.Type.GAUGE, stream);
+
+ // generate managerLedger metrics
+ parseMetricsToPrometheusMetrics(new ManagedLedgerMetrics(pulsar).generate(),
+ clusterName, "manageLedger", Collector.Type.GAUGE, stream);
+
+ // generate brokerService topic metrics
+ parseMetricsToPrometheusMetrics(pulsar.getBrokerService().getTopicMetrics(),
+ clusterName, "brokerService", Collector.Type.GAUGE, stream);
Review comment:
We already expose topic level metrics, why we need to add topic metrics again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services