You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/05/12 08:16:20 UTC
[GitHub] [pulsar] marksilcox opened a new pull request, #15558: [fix][broker] Ensure prometheus metrics are grouped by type (#8407)
marksilcox opened a new pull request, #15558:
URL: https://github.com/apache/pulsar/pull/15558
<!--
### Contribution Checklist
- PR title format should be *[type][component] summary*. For details, see *[Guideline - Pulsar PR Naming Convention](https://docs.google.com/document/d/1d8Pw6ZbWk-_pCKdOmdvx9rnhPiyuxwq60_TrD68d7BA/edit#heading=h.trs9rsex3xom)*.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Each pull request should address only one issue, not mix up code from multiple issues.
- Each commit in the pull request has a meaningful commit message
- Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
**(The sections below can be removed for hotfixes of typos)**
-->
Fixes #8407
### Motivation
Current broker prometheus metrics are not grouped by metric type which causes issues in systems that read these metrics (e.g. DataDog).
Prometheus docs states "All lines for a given metric must be provided as one single group" - https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md#grouping-and-sorting
Modifications
Updated the namespace and topic prometheus metric generators to group the metrics under the appropriate type header.
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
This change added tests and can be verified as follows:
Added unit test to verify all metrics are grouped under correct type header
### Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API: (no)
- The schema: (no)
- The default values of configurations: (no)
- The wire protocol: (no)
- The rest endpoints: (no)
- The admin cli options: (no)
- Anything that affects deployment: (no)
### Documentation
Need to update docs?
- [X] `no-need-doc`
Changes to match prometheus spec
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] tjiuming commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
tjiuming commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1186571691
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on a diff in pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on code in PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#discussion_r913078015
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/NamespaceStatsAggregator.java:
##########
@@ -44,62 +51,65 @@
@Slf4j
public class NamespaceStatsAggregator {
- private static FastThreadLocal<AggregatedNamespaceStats> localNamespaceStats =
- new FastThreadLocal<AggregatedNamespaceStats>() {
+ private static final FastThreadLocal<AggregatedNamespaceStats> localNamespaceStats =
+ new FastThreadLocal<>() {
@Override
- protected AggregatedNamespaceStats initialValue() throws Exception {
+ protected AggregatedNamespaceStats initialValue() {
return new AggregatedNamespaceStats();
}
};
- private static FastThreadLocal<TopicStats> localTopicStats = new FastThreadLocal<TopicStats>() {
+ private static final FastThreadLocal<TopicStats> localTopicStats = new FastThreadLocal<>() {
@Override
- protected TopicStats initialValue() throws Exception {
+ protected TopicStats initialValue() {
return new TopicStats();
}
};
public static void generate(PulsarService pulsar, boolean includeTopicMetrics, boolean includeConsumerMetrics,
- boolean includeProducerMetrics, boolean splitTopicAndPartitionIndexLabel, SimpleTextOutputStream stream) {
+ boolean includeProducerMetrics, boolean splitTopicAndPartitionIndexLabel,
+ SimpleTextOutputStream stream) {
String cluster = pulsar.getConfiguration().getClusterName();
AggregatedNamespaceStats namespaceStats = localNamespaceStats.get();
- TopicStats.resetTypes();
TopicStats topicStats = localTopicStats.get();
-
- printDefaultBrokerStats(stream, cluster);
-
Optional<CompactorMXBean> compactorMXBean = getCompactorMXBean(pulsar);
LongAdder topicsCount = new LongAdder();
+ Map<String, ByteBuf> allMetrics = new HashMap<>();
Review Comment:
We would however still need to have some access to the `ByteBuf`'s to be able to release them?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1137151554
/pulsarbot run cpp-tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1136910704
/pulsarbot run cpp-tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1233086664
@Jason918
Running the same suite of tests (`./build/run_integration_group.sh PULSAR_CONNECTORS_THREAD`) passes locally for me (after updating with `master`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] asafm commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
asafm commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1217667451
Just need to see how to get those tests green. @nlu90 How can we move forward?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
liangyepianzhou commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1350340448
I move this to 2.10.4. If you have any questions, feel free to ping me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] asafm commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
asafm commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1162169821
I definitely agree that for the developer you'll be spending more time writing this and checking for collisions, and maybe printing from one class one variable and then another class same variable, but you will obtain the end result of not increasing memory allocations. The more I read into Pulsar metrics code, the more I see techniques used to decrease memory allocations. Just today I saw in TransactionsAggregator that they avoid allocating objects and keeping the object in ThreadLocal and reset it every time they need it, like object pool, but with a fixed size (number of threads).
I guess @merlimat and @codelipenghui can contribute their knowledge here on how important that is, as I'm fairly new to this project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on a diff in pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on code in PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#discussion_r908213548
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java:
##########
@@ -1501,6 +1503,59 @@ private void compareCompactionStateCount(List<Metric> cm, double count) {
assertEquals(cm.get(0).value, count);
}
+ @Test
+ public void testMetricsGroupedByTypeDefinitions() throws Exception {
+ Producer<byte[]> p1 = pulsarClient.newProducer().topic("persistent://my-property/use/my-ns/my-topic1").create();
+ Producer<byte[]> p2 = pulsarClient.newProducer().topic("persistent://my-property/use/my-ns/my-topic2").create();
+ for (int i = 0; i < 10; i++) {
+ String message = "my-message-" + i;
+ p1.send(message.getBytes());
+ p2.send(message.getBytes());
+ }
+
+ ByteArrayOutputStream statsOut = new ByteArrayOutputStream();
+ PrometheusMetricsGenerator.generate(pulsar, false, false, false, statsOut);
+ String metricsStr = statsOut.toString();
+
+ Pattern typePattern = Pattern.compile("^#\\s+TYPE\\s+(\\w+)\\s+(\\w+)");
+ Pattern metricNamePattern = Pattern.compile("^(\\w+)\\{.+");
+
+ AtomicReference<String> currentMetric = new AtomicReference<>();
+ Splitter.on("\n").split(metricsStr).forEach(line -> {
+ if (line.isEmpty()) {
+ return;
+ }
+ if (line.startsWith("#")) {
+ // Get the current type definition
+ Matcher typeMatcher = typePattern.matcher(line);
+ checkArgument(typeMatcher.matches());
+ String metricName = typeMatcher.group(1);
+ currentMetric.set(metricName);
+ } else {
+ Matcher metricMatcher = metricNamePattern.matcher(line);
+ checkArgument(metricMatcher.matches());
+ String metricName = metricMatcher.group(1);
+
+ if (metricName.endsWith("_bucket")) {
+ metricName = metricName.substring(0, metricName.indexOf("_bucket"));
+ } else if (metricName.endsWith("_count") && !currentMetric.get().endsWith("_count")) {
+ metricName = metricName.substring(0, metricName.indexOf("_count"));
+ } else if (metricName.endsWith("_sum") && !currentMetric.get().endsWith("_sum")) {
+ metricName = metricName.substring(0, metricName.indexOf("_sum"));
+ }
+
+ if (!metricName.equals(currentMetric.get())) {
+ System.out.println(metricsStr);
Review Comment:
following a pattern already in this test of using sysout
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
michaeljmarshall commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1241491769
Thanks for clarifying @marksilcox. I misunderstood the initial issue. This seems like a bug fix, not an improvement @eolivelli.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jason918 commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
Jason918 commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1232403191
> @codelipenghui @eolivelli Let's merge this?
We can't. The required CI still fails. I restarted it for the third time.
If it still fails. I think @marksilcox should take a deeper look at it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] asafm commented on a diff in pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
asafm commented on code in PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#discussion_r898048085
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/metrics/PrometheusTextFormatUtil.java:
##########
@@ -141,29 +140,32 @@ private static void writeSum(Writer w, DataSketchesOpStatsLogger opStat, String
.append(Double.toString(opStat.getSum(success))).append('\n');
}
- public static void writeMetricsCollectedByPrometheusClient(Writer w, CollectorRegistry registry)
- throws IOException {
- Enumeration<MetricFamilySamples> metricFamilySamples = registry.metricFamilySamples();
- while (metricFamilySamples.hasMoreElements()) {
- MetricFamilySamples metricFamily = metricFamilySamples.nextElement();
-
- for (int i = 0; i < metricFamily.samples.size(); i++) {
- Sample sample = metricFamily.samples.get(i);
- w.write(sample.name);
- w.write('{');
- for (int j = 0; j < sample.labelNames.size(); j++) {
- if (j != 0) {
- w.write(", ");
+ public static void writeMetrics(SimpleTextOutputStream stream, Collection<MetricFamilySamples> familySamples) {
+ for (MetricFamilySamples familySample : familySamples) {
+ stream.write("# TYPE ");
+ stream.write(familySample.name);
+ stream.write(' ');
+ stream.write(familySample.type.name().toLowerCase());
+ stream.write('\n');
+ for (Collector.MetricFamilySamples.Sample sample : familySample.samples) {
+ stream.write(sample.name);
+ if (sample.labelNames.size() > 0) {
+ stream.write('{');
+ for (int i = 0; i < sample.labelNames.size(); ++i) {
+ stream.write(sample.labelNames.get(i));
+ stream.write("=\"");
+ stream.write(sample.labelValues.get(i));
+ stream.write("\",");
}
- w.write(sample.labelNames.get(j));
- w.write("=\"");
- w.write(sample.labelValues.get(j));
- w.write('"');
+ stream.write('}');
}
-
- w.write("} ");
- w.write(Collector.doubleToGoString(sample.value));
- w.write('\n');
+ stream.write(' ');
+ stream.write(sample.value);
Review Comment:
Don't forget to use the method used previously
``` /**
* Convert a double to its string representation in Go.
*/
public static String doubleToGoString(double d) {
if (d == Double.POSITIVE_INFINITY) {
return "+Inf";
}
if (d == Double.NEGATIVE_INFINITY) {
return "-Inf";
}
if (Double.isNaN(d)) {
return "NaN";
}
return Double.toString(d);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on a diff in pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on code in PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#discussion_r936394736
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricStreams.java:
##########
@@ -0,0 +1,74 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.stats.prometheus;
+
+import io.netty.buffer.ByteBufAllocator;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pulsar.common.util.SimpleTextOutputStream;
+
+/**
+ * Helper class to ensure that metrics of the same name are grouped together under the same TYPE header when written.
+ */
+public class PrometheusMetricStreams {
+ private final Map<String, SimpleTextOutputStream> metricStreamMap = new HashMap<>();
+
+ /**
+ * Write the given metric and sample value to the stream. Will write #TYPE header if metric not seen before.
+ * @param metricName name of the metric.
+ * @param value value of the sample
+ * @param labelsAndValuesArray varargs of label and label value
+ */
+ void writeSample(String metricName, Number value, String... labelsAndValuesArray) {
+ SimpleTextOutputStream stream = initGaugeType(metricName);
+ stream.write(metricName).write('{');
+ for (int i = 0; i < labelsAndValuesArray.length; i += 2) {
+ stream.write(labelsAndValuesArray[i]).write("=\"").write(labelsAndValuesArray[i + 1]).write('\"');
+ if (i + 2 != labelsAndValuesArray.length) {
+ stream.write(',');
+ }
+ }
+ stream.write("\"} ").write(value).write(' ').write(System.currentTimeMillis()).write('\n');
Review Comment:
@asafm @hangc0276 @merlimat @tjiuming
Assume removing of timestamp would be under a different PR as this one is specifically about grouping of metrics and removing of timestamp would affect a lot more metrics than the ones this PR touches.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on a diff in pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on code in PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#discussion_r872496951
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/TopicStats.java:
##########
@@ -382,77 +389,102 @@ static void metricType(SimpleTextOutputStream stream, String name) {
}
- private static void metric(SimpleTextOutputStream stream, String cluster, String namespace, String topic,
- String name, double value, boolean splitTopicAndPartitionIndexLabel) {
- metricType(stream, name);
- appendRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel).write("\"} ");
- stream.write(value);
- appendEndings(stream);
+ private static void metric(Map<String, List<String>> metrics, String cluster, String namespace, String topic,
+ String name, double value, boolean splitTopicAndPartitionIndexLabel) {
+ writeMetric(metrics, name, stream -> {
+ writeRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel);
+ stream.write("\"} ")
+ .write(value);
+ writeEndings(stream);
+ });
}
- private static void metric(SimpleTextOutputStream stream, String cluster, String namespace, String topic,
- String subscription, String name, long value, boolean splitTopicAndPartitionIndexLabel) {
- metricType(stream, name);
- appendRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel)
- .write("\",subscription=\"").write(subscription).write("\"} ");
- stream.write(value);
- appendEndings(stream);
+ private static void metric(Map<String, List<String>> metrics, String cluster, String namespace, String topic,
+ String subscription, String name, long value, boolean splitTopicAndPartitionIndexLabel) {
+ writeMetric(metrics, name, stream -> {
+ writeRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel);
+ stream.write("\",subscription=\"").write(subscription)
+ .write("\"} ")
+ .write(value);
+ writeEndings(stream);
+ });
+ }
+
+ private static void metric(Map<String, List<String>> metrics, String cluster, String namespace, String topic,
+ String producerName, long produceId, String name, double value,
+ boolean splitTopicAndPartitionIndexLabel) {
+ writeMetric(metrics, name, stream -> {
+ writeRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel);
+ stream.write("\",producer_name=\"").write(producerName)
+ .write("\",producer_id=\"").write(produceId)
+ .write("\"} ")
+ .write(value);
+ writeEndings(stream);
+ });
}
- private static void metric(SimpleTextOutputStream stream, String cluster, String namespace, String topic,
- String producerName, long produceId, String name, double value, boolean splitTopicAndPartitionIndexLabel) {
- metricType(stream, name);
- appendRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel)
- .write("\",producer_name=\"").write(producerName)
- .write("\",producer_id=\"").write(produceId).write("\"} ");
- stream.write(value);
- appendEndings(stream);
+ private static void metric(Map<String, List<String>> metrics, String cluster, String namespace, String topic,
+ String subscription, String name, double value,
+ boolean splitTopicAndPartitionIndexLabel) {
+ writeMetric(metrics, name, stream -> {
+ writeRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel);
+ stream.write("\",subscription=\"").write(subscription)
+ .write("\"} ")
+ .write(value);
+ writeEndings(stream);
+ });
}
- private static void metric(SimpleTextOutputStream stream, String cluster, String namespace, String topic,
- String subscription, String name, double value, boolean splitTopicAndPartitionIndexLabel) {
- metricType(stream, name);
- appendRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel)
- .write("\",subscription=\"").write(subscription).write("\"} ");
- stream.write(value);
- appendEndings(stream);
+ private static void metric(Map<String, List<String>> metrics, String cluster, String namespace, String topic,
+ String subscription, String consumerName, long consumerId, String name, long value,
+ boolean splitTopicAndPartitionIndexLabel) {
+ writeMetric(metrics, name, stream -> {
+ writeRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel);
+ stream.write("\",subscription=\"").write(subscription)
+ .write("\",consumer_name=\"").write(consumerName).write("\",consumer_id=\"").write(consumerId)
+ .write("\"} ")
+ .write(value);
+ writeEndings(stream);
+ });
}
- private static void metric(SimpleTextOutputStream stream, String cluster, String namespace, String topic,
- String subscription, String consumerName, long consumerId, String name, long value,
- boolean splitTopicAndPartitionIndexLabel) {
- metricType(stream, name);
- appendRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel)
- .write("\",subscription=\"").write(subscription)
- .write("\",consumer_name=\"").write(consumerName).write("\",consumer_id=\"").write(consumerId)
- .write("\"} ");
- stream.write(value);
- appendEndings(stream);
+ private static void metric(Map<String, List<String>> metrics, String cluster, String namespace, String topic,
+ String subscription, String consumerName, long consumerId, String name, double value,
+ boolean splitTopicAndPartitionIndexLabel) {
+ writeMetric(metrics, name, stream -> {
+ writeRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel);
+ stream.write("\",subscription=\"").write(subscription)
+ .write("\",consumer_name=\"").write(consumerName).write("\",consumer_id=\"")
+ .write(consumerId).write("\"} ")
+ .write(value);
+ writeEndings(stream);
+ });
}
- private static void metric(SimpleTextOutputStream stream, String cluster, String namespace, String topic,
- String subscription, String consumerName, long consumerId, String name, double value,
- boolean splitTopicAndPartitionIndexLabel) {
- metricType(stream, name);
- appendRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel)
- .write("\",subscription=\"").write(subscription)
- .write("\",consumer_name=\"").write(consumerName).write("\",consumer_id=\"")
- .write(consumerId).write("\"} ");
- stream.write(value);
- appendEndings(stream);
+ private static void metricWithRemoteCluster(Map<String, List<String>> metrics, String cluster, String namespace,
+ String topic, String name, String remoteCluster, double value,
+ boolean splitTopicAndPartitionIndexLabel) {
+ writeMetric(metrics, name, stream -> {
+ writeRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel);
+ stream.write("\",remote_cluster=\"").write(remoteCluster).write("\"} ").write(value);
+ writeEndings(stream);
+ });
}
- private static void metricWithRemoteCluster(SimpleTextOutputStream stream, String cluster, String namespace,
- String topic, String name, String remoteCluster, double value, boolean splitTopicAndPartitionIndexLabel) {
- metricType(stream, name);
- appendRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel)
- .write("\",remote_cluster=\"").write(remoteCluster).write("\"} ");
- stream.write(value);
- appendEndings(stream);
+ private static void writeMetric(Map<String, List<String>> metrics, String name,
+ Consumer<SimpleTextOutputStream> metricWriter) {
+ ByteBuf buf = ByteBufAllocator.DEFAULT.heapBuffer();
Review Comment:
agree - have updated to use prometheus' `Collector.MetricFamilySamples` with only one `SimpleTextOutputStream` used when writing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
eolivelli commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1233978863
> @Jason918 those 2 failing tests are passing when I run them locally. Not sure how to get these unit tests to pass on GitHub other than keep re running until they do!
I have restarted the failing jobs. the failures are not related to this patch
I will commit as soon as CI passes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jason918 commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
Jason918 commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1236835633
@marksilcox Can you help open a new PR to branch-2.10? There are a lot conflict when cherry-pick directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jason918 commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
Jason918 commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1235234677
@marksilcox Merged, thanks for your contribution!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on a diff in pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on code in PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#discussion_r914840072
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricsGenerator.java:
##########
@@ -181,22 +181,26 @@ private static ByteBuf generate0(PulsarService pulsar, boolean includeTopicMetri
//when write out 200MB data, MAX_COMPONENTS = 64 needn't mem_copy. see: CompositeByteBuf#consolidateIfNeeded()
ByteBuf buf = UnpooledByteBufAllocator.DEFAULT.compositeDirectBuffer(MAX_COMPONENTS);
boolean exceptionHappens = false;
+ //Used in namespace/topic and transaction aggregators as share metric names
Review Comment:
Some metric names used in `TransactionAggregator` and `TopicStats` are the same - e.g. `pulsar_storage_ledger_write_latency_le_0_5` - I don't see anywhere these are prefixed differently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] asafm commented on a diff in pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
asafm commented on code in PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#discussion_r913084580
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/NamespaceStatsAggregator.java:
##########
@@ -44,62 +51,65 @@
@Slf4j
public class NamespaceStatsAggregator {
- private static FastThreadLocal<AggregatedNamespaceStats> localNamespaceStats =
- new FastThreadLocal<AggregatedNamespaceStats>() {
+ private static final FastThreadLocal<AggregatedNamespaceStats> localNamespaceStats =
+ new FastThreadLocal<>() {
@Override
- protected AggregatedNamespaceStats initialValue() throws Exception {
+ protected AggregatedNamespaceStats initialValue() {
return new AggregatedNamespaceStats();
}
};
- private static FastThreadLocal<TopicStats> localTopicStats = new FastThreadLocal<TopicStats>() {
+ private static final FastThreadLocal<TopicStats> localTopicStats = new FastThreadLocal<>() {
@Override
- protected TopicStats initialValue() throws Exception {
+ protected TopicStats initialValue() {
return new TopicStats();
}
};
public static void generate(PulsarService pulsar, boolean includeTopicMetrics, boolean includeConsumerMetrics,
- boolean includeProducerMetrics, boolean splitTopicAndPartitionIndexLabel, SimpleTextOutputStream stream) {
+ boolean includeProducerMetrics, boolean splitTopicAndPartitionIndexLabel,
+ SimpleTextOutputStream stream) {
String cluster = pulsar.getConfiguration().getClusterName();
AggregatedNamespaceStats namespaceStats = localNamespaceStats.get();
- TopicStats.resetTypes();
TopicStats topicStats = localTopicStats.get();
-
- printDefaultBrokerStats(stream, cluster);
-
Optional<CompactorMXBean> compactorMXBean = getCompactorMXBean(pulsar);
LongAdder topicsCount = new LongAdder();
+ Map<String, ByteBuf> allMetrics = new HashMap<>();
Review Comment:
@marksilcox Yes, you are correct. Maybe add `getByteBuf()` to `SimpleTextOutputStream` and then flush it and then release. Another option is keeping another map to ByteBuf, but I'm not sure it's so elegant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on a diff in pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on code in PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#discussion_r913006917
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/NamespaceStatsAggregator.java:
##########
@@ -296,161 +306,264 @@ private static void getTopicStats(Topic topic, TopicStats stats, boolean include
});
}
- private static void printDefaultBrokerStats(SimpleTextOutputStream stream, String cluster) {
- // Print metrics with 0 values. This is necessary to have the available brokers being
- // reported in the brokers dashboard even if they don't have any topic or traffic
- metric(stream, cluster, "pulsar_topics_count", 0);
- metric(stream, cluster, "pulsar_subscriptions_count", 0);
- metric(stream, cluster, "pulsar_producers_count", 0);
- metric(stream, cluster, "pulsar_consumers_count", 0);
- metric(stream, cluster, "pulsar_rate_in", 0);
- metric(stream, cluster, "pulsar_rate_out", 0);
- metric(stream, cluster, "pulsar_throughput_in", 0);
- metric(stream, cluster, "pulsar_throughput_out", 0);
- metric(stream, cluster, "pulsar_storage_size", 0);
- metric(stream, cluster, "pulsar_storage_logical_size", 0);
- metric(stream, cluster, "pulsar_storage_write_rate", 0);
- metric(stream, cluster, "pulsar_storage_read_rate", 0);
- metric(stream, cluster, "pulsar_msg_backlog", 0);
+ private static void printTopicsCountStats(Map<String, ByteBuf> allMetrics, Map<String, Long> namespaceTopicsCount,
+ String cluster) {
+ namespaceTopicsCount.forEach((ns, topicCount) ->
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_topics_count", topicCount, cluster, ns)
+ );
}
- private static void printTopicsCountStats(SimpleTextOutputStream stream, String cluster, String namespace,
- LongAdder topicsCount) {
- metric(stream, cluster, namespace, "pulsar_topics_count", topicsCount.sum());
- }
+ private static void printNamespaceStats(Map<String, ByteBuf> allMetrics, AggregatedNamespaceStats stats,
+ String cluster, String namespace) {
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_topics_count", stats.topicsCount, cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_subscriptions_count", stats.subscriptionsCount, cluster,
+ namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_producers_count", stats.producersCount, cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_consumers_count", stats.consumersCount, cluster, namespace);
+
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_rate_in", stats.rateIn, cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_rate_out", stats.rateOut, cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_throughput_in", stats.throughputIn, cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_throughput_out", stats.throughputOut, cluster, namespace);
+ writeMetric(allMetrics, "pulsar_consumer_msg_ack_rate", stats.messageAckRate, cluster, namespace);
+
+ writeMetric(allMetrics, "pulsar_in_bytes_total", stats.bytesInCounter, cluster, namespace);
+ writeMetric(allMetrics, "pulsar_in_messages_total", stats.msgInCounter, cluster, namespace);
+ writeMetric(allMetrics, "pulsar_out_bytes_total", stats.bytesOutCounter, cluster, namespace);
+ writeMetric(allMetrics, "pulsar_out_messages_total", stats.msgOutCounter, cluster, namespace);
+
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_storage_size", stats.managedLedgerStats.storageSize, cluster,
+ namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_storage_logical_size",
+ stats.managedLedgerStats.storageLogicalSize, cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_backlog_size", stats.managedLedgerStats.backlogSize, cluster,
+ namespace);
+ writeMetric(allMetrics, "pulsar_storage_offloaded_size",
+ stats.managedLedgerStats.offloadedStorageUsed, cluster, namespace);
+
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_storage_write_rate", stats.managedLedgerStats.storageWriteRate,
+ cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_storage_read_rate", stats.managedLedgerStats.storageReadRate,
+ cluster, namespace);
+
+ writeMetric(allMetrics, "pulsar_subscription_delayed", stats.msgDelayed, cluster, namespace);
+
+ writePulsarMsgBacklog(allMetrics, stats.msgBacklog, cluster, namespace);
- private static void printNamespaceStats(SimpleTextOutputStream stream, String cluster, String namespace,
- AggregatedNamespaceStats stats) {
- metric(stream, cluster, namespace, "pulsar_topics_count", stats.topicsCount);
- metric(stream, cluster, namespace, "pulsar_subscriptions_count", stats.subscriptionsCount);
- metric(stream, cluster, namespace, "pulsar_producers_count", stats.producersCount);
- metric(stream, cluster, namespace, "pulsar_consumers_count", stats.consumersCount);
+ stats.managedLedgerStats.storageWriteLatencyBuckets.refresh();
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_0_5",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[0], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_1",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[1], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_5",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[2], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_10",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[3], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_20",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[4], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_50",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[5], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_100",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[6], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_200",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[7], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_1000",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[8], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_overflow",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[9], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_count",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getCount(), cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_sum",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getSum(), cluster, namespace);
- metric(stream, cluster, namespace, "pulsar_rate_in", stats.rateIn);
- metric(stream, cluster, namespace, "pulsar_rate_out", stats.rateOut);
- metric(stream, cluster, namespace, "pulsar_throughput_in", stats.throughputIn);
- metric(stream, cluster, namespace, "pulsar_throughput_out", stats.throughputOut);
- metric(stream, cluster, namespace, "pulsar_consumer_msg_ack_rate", stats.messageAckRate);
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.refresh();
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_0_5",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[0], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_1",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[1], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_5",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[2], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_10",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[3], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_20",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[4], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_50",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[5], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_100",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[6], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_200",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[7], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_1000",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[8], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_overflow",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[9], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_count",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getCount(), cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_sum",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getSum(), cluster, namespace);
- metric(stream, cluster, namespace, "pulsar_in_bytes_total", stats.bytesInCounter);
- metric(stream, cluster, namespace, "pulsar_in_messages_total", stats.msgInCounter);
- metric(stream, cluster, namespace, "pulsar_out_bytes_total", stats.bytesOutCounter);
- metric(stream, cluster, namespace, "pulsar_out_messages_total", stats.msgOutCounter);
+ stats.managedLedgerStats.entrySizeBuckets.refresh();
+ writeMetric(allMetrics, "pulsar_entry_size_le_128", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[0],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_512", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[1],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_1_kb", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[2],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_2_kb", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[3],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_4_kb", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[4],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_16_kb", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[5],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_100_kb",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[6], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_1_mb", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[7],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_overflow",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[8], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_count",
+ stats.managedLedgerStats.entrySizeBuckets.getCount(), cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_sum",
+ stats.managedLedgerStats.entrySizeBuckets.getSum(), cluster, namespace);
+
+ writeReplicationStat(allMetrics, "pulsar_replication_rate_in", stats,
+ replStats -> replStats.msgRateIn, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_rate_out", stats,
+ replStats -> replStats.msgRateOut, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_throughput_in", stats,
+ replStats -> replStats.msgThroughputIn, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_throughput_out", stats,
+ replStats -> replStats.msgThroughputOut, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_backlog", stats,
+ replStats -> replStats.replicationBacklog, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_connected_count", stats,
+ replStats -> replStats.connectedCount, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_rate_expired", stats,
+ replStats -> replStats.msgRateExpired, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_delay_in_seconds", stats,
+ replStats -> replStats.replicationDelayInSeconds, cluster, namespace);
+ }
- metric(stream, cluster, namespace, "pulsar_storage_size", stats.managedLedgerStats.storageSize);
- metric(stream, cluster, namespace, "pulsar_storage_logical_size", stats.managedLedgerStats.storageLogicalSize);
- metric(stream, cluster, namespace, "pulsar_storage_backlog_size", stats.managedLedgerStats.backlogSize);
- metric(stream, cluster, namespace, "pulsar_storage_offloaded_size",
- stats.managedLedgerStats.offloadedStorageUsed);
+ private static void writeMetricWithBrokerDefault(Map<String, ByteBuf> allMetrics, String metricName, Number value,
+ String cluster, String namespace) {
+ ByteBuf buffer = writeGaugeTypeWithBrokerDefault(allMetrics, metricName, cluster);
+ writeSample(buffer, metricName, value, "cluster", cluster, "namespace", namespace);
+ }
- metric(stream, cluster, namespace, "pulsar_storage_write_rate", stats.managedLedgerStats.storageWriteRate);
- metric(stream, cluster, namespace, "pulsar_storage_read_rate", stats.managedLedgerStats.storageReadRate);
+ private static void writePulsarMsgBacklog(Map<String, ByteBuf> allMetrics, Number value,
+ String cluster, String namespace) {
+ ByteBuf buffer = writeGaugeTypeWithBrokerDefault(allMetrics, "pulsar_msg_backlog", cluster);
+ writeSample(buffer, "pulsar_msg_backlog", value, "cluster", cluster, "namespace", namespace, "remote_cluster",
+ "local");
+ }
- metric(stream, cluster, namespace, "pulsar_subscription_delayed", stats.msgDelayed);
+ private static void writeMetric(Map<String, ByteBuf> allMetrics, String metricName, Number value, String cluster,
+ String namespace) {
+ ByteBuf buffer = writeGaugeType(allMetrics, metricName);
+ writeSample(buffer, metricName, value, "cluster", cluster, "namespace", namespace);
+ }
- metricWithRemoteCluster(stream, cluster, namespace, "pulsar_msg_backlog", "local", stats.msgBacklog);
+ private static void writeReplicationStat(Map<String, ByteBuf> allMetrics, String metricName,
+ AggregatedNamespaceStats namespaceStats,
+ Function<AggregatedReplicationStats, Number> replStatsFunction,
+ String cluster, String namespace) {
+ if (!namespaceStats.replicationStats.isEmpty()) {
+ ByteBuf buffer = writeGaugeType(allMetrics, metricName);
+ namespaceStats.replicationStats.forEach((remoteCluster, replStats) ->
+ writeSample(buffer, metricName, replStatsFunction.apply(replStats),
+ "cluster", cluster,
+ "namespace", namespace,
+ "remote_cluster", remoteCluster)
+ );
+ }
+ }
- stats.managedLedgerStats.storageWriteLatencyBuckets.refresh();
- long[] latencyBuckets = stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets();
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_0_5", latencyBuckets[0]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_1", latencyBuckets[1]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_5", latencyBuckets[2]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_10", latencyBuckets[3]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_20", latencyBuckets[4]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_50", latencyBuckets[5]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_100", latencyBuckets[6]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_200", latencyBuckets[7]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_1000", latencyBuckets[8]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_overflow", latencyBuckets[9]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_count",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getCount());
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_sum",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getSum());
+ static ByteBuf writeGaugeType(Map<String, ByteBuf> allMetrics, String metricName) {
+ if (!allMetrics.containsKey(metricName)) {
+ ByteBuf buffer = UnpooledByteBufAllocator.DEFAULT.compositeDirectBuffer(MAX_COMPONENTS);
Review Comment:
copied from `PrometheusMetricsGenerator` to get me started and forgot to replace - replaced with `ByteBufAllocator.DEFAULT.heapBuffer()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] asafm commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
asafm commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1217666597
@marksilcox Your work was great. We can settle for opening an issue and linking this PR to it - what do you say?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on a diff in pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on code in PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#discussion_r913005808
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/NamespaceStatsAggregator.java:
##########
@@ -296,161 +306,264 @@ private static void getTopicStats(Topic topic, TopicStats stats, boolean include
});
}
- private static void printDefaultBrokerStats(SimpleTextOutputStream stream, String cluster) {
- // Print metrics with 0 values. This is necessary to have the available brokers being
- // reported in the brokers dashboard even if they don't have any topic or traffic
- metric(stream, cluster, "pulsar_topics_count", 0);
- metric(stream, cluster, "pulsar_subscriptions_count", 0);
- metric(stream, cluster, "pulsar_producers_count", 0);
- metric(stream, cluster, "pulsar_consumers_count", 0);
- metric(stream, cluster, "pulsar_rate_in", 0);
- metric(stream, cluster, "pulsar_rate_out", 0);
- metric(stream, cluster, "pulsar_throughput_in", 0);
- metric(stream, cluster, "pulsar_throughput_out", 0);
- metric(stream, cluster, "pulsar_storage_size", 0);
- metric(stream, cluster, "pulsar_storage_logical_size", 0);
- metric(stream, cluster, "pulsar_storage_write_rate", 0);
- metric(stream, cluster, "pulsar_storage_read_rate", 0);
- metric(stream, cluster, "pulsar_msg_backlog", 0);
+ private static void printTopicsCountStats(Map<String, ByteBuf> allMetrics, Map<String, Long> namespaceTopicsCount,
+ String cluster) {
+ namespaceTopicsCount.forEach((ns, topicCount) ->
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_topics_count", topicCount, cluster, ns)
Review Comment:
replaced `printDefaultBrokerStats`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1190033031
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] nahguam commented on a diff in pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
nahguam commented on code in PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#discussion_r908193576
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java:
##########
@@ -1501,6 +1503,59 @@ private void compareCompactionStateCount(List<Metric> cm, double count) {
assertEquals(cm.get(0).value, count);
}
+ @Test
+ public void testMetricsGroupedByTypeDefinitions() throws Exception {
+ Producer<byte[]> p1 = pulsarClient.newProducer().topic("persistent://my-property/use/my-ns/my-topic1").create();
+ Producer<byte[]> p2 = pulsarClient.newProducer().topic("persistent://my-property/use/my-ns/my-topic2").create();
+ for (int i = 0; i < 10; i++) {
+ String message = "my-message-" + i;
+ p1.send(message.getBytes());
+ p2.send(message.getBytes());
+ }
+
+ ByteArrayOutputStream statsOut = new ByteArrayOutputStream();
+ PrometheusMetricsGenerator.generate(pulsar, false, false, false, statsOut);
+ String metricsStr = statsOut.toString();
+
+ Pattern typePattern = Pattern.compile("^#\\s+TYPE\\s+(\\w+)\\s+(\\w+)");
+ Pattern metricNamePattern = Pattern.compile("^(\\w+)\\{.+");
+
+ AtomicReference<String> currentMetric = new AtomicReference<>();
+ Splitter.on("\n").split(metricsStr).forEach(line -> {
+ if (line.isEmpty()) {
+ return;
+ }
+ if (line.startsWith("#")) {
+ // Get the current type definition
+ Matcher typeMatcher = typePattern.matcher(line);
+ checkArgument(typeMatcher.matches());
+ String metricName = typeMatcher.group(1);
+ currentMetric.set(metricName);
+ } else {
+ Matcher metricMatcher = metricNamePattern.matcher(line);
+ checkArgument(metricMatcher.matches());
+ String metricName = metricMatcher.group(1);
+
+ if (metricName.endsWith("_bucket")) {
+ metricName = metricName.substring(0, metricName.indexOf("_bucket"));
+ } else if (metricName.endsWith("_count") && !currentMetric.get().endsWith("_count")) {
+ metricName = metricName.substring(0, metricName.indexOf("_count"));
+ } else if (metricName.endsWith("_sum") && !currentMetric.get().endsWith("_sum")) {
+ metricName = metricName.substring(0, metricName.indexOf("_sum"));
+ }
+
+ if (!metricName.equals(currentMetric.get())) {
+ System.out.println(metricsStr);
+ fail("Metric not grouped under it's type definition: " + line);
Review Comment:
```suggestion
fail("Metric not grouped under its type definition: " + line);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1163178010
The problem we have is that we need to collect by metric name, and not by topic/namespace as is currently happening. So for a metric such as `pulsar_rate_in` whether `includeTopics` is enabled or not we need all values of this metric grouped under a single type header.
This means whatever is used needs to store each metric in a way so other namespaces or topics can add there values. I guess what I need to find is the most memory efficient way of managing this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
eolivelli commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1217672058
/pulsar-bot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1208296820
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1352778289
@liangyepianzhou I've created https://github.com/apache/pulsar/pull/18941 to merge into branch-2.10.
@asafm compared these changes against that PR to ensure the memory leak fix is in place
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] asafm commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
asafm commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1161898564
@tjiuming Maybe you meant @marksilcox ?
I have a note regarding the way we write `NamespaceStatsAggregator`.
The whole idea in the way the metric system is built today in Pulsar (I know it's fractured) is to try to avoid memory allocation as much as possible. It does so by:
* Using `long` and `AtomicLongFieldUpdater` instead of `AtomicLong`
* Writing the values directly into ByteBuf and then to the stream.
* In some cases directly into the stream
In your refactor, we add a Sample object each containing 2 lists and a value (Double). On the way, we create a Map, and of course a `MetricFamilySamples` object. The sample is done per metrics we have.
I'm just starting to learn the metrics code in Pulsar, but I think it beats the point of the original code. I believe @merlimat can correct me.
What I suggest is to build the code to achieve the goal you desire, each sample belonging to a specific metric will be written one after another.
How?
Let's collect all `TopicStats`. We pay the cost of allocating them anyway. Once we have that, we can write a function writing each individual metric samples. Each will call a function that looks like:
`writeMetric(stream, "msgsRate", cluster, namespace, topicToStatsMap, topicStats -> topicStats.msgsRate)`
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1233963404
@Jason918 those 2 failing tests are passing when I run them locally. Not sure how to get these unit tests to pass on GitHub other than keep re running until they do!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1233338601
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jason918 commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
Jason918 commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1236835392
> Is there a reason we don't have this labeled for 2.8.5? I opened #17419 and would like to cherry pick it to 2.8.5. Cherry picking that will be much easier if this can get cherry picked first.
@codelipenghui ping
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jason918 commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
Jason918 commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1238870295
> branch-2.10 must be kept stable and we should cherry-pick only stuff that we are sure it is not going to break stability or compatibility.
I agree absolutely, there are over 190 commits included in 2.10.2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] asafm commented on a diff in pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
asafm commented on code in PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#discussion_r908171501
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/NamespaceStatsAggregator.java:
##########
@@ -68,46 +82,59 @@ public static void generate(PulsarService pulsar, boolean includeTopicMetrics, b
boolean includeProducerMetrics, boolean splitTopicAndPartitionIndexLabel,
SimpleTextOutputStream stream) {
String cluster = pulsar.getConfiguration().getClusterName();
- AggregatedNamespaceStats namespaceStats = localNamespaceStats.get();
- TopicStats.resetTypes();
- TopicStats topicStats = localTopicStats.get();
-
- Map<String, Collector.MetricFamilySamples> metrics = new HashMap<>();
-
- buildDefaultBrokerStats(metrics, cluster);
-
Optional<CompactorMXBean> compactorMXBean = getCompactorMXBean(pulsar);
LongAdder topicsCount = new LongAdder();
- pulsar.getBrokerService().getMultiLayerTopicMap().forEach((namespace, bundlesMap) -> {
- namespaceStats.reset();
- topicsCount.reset();
-
- bundlesMap.forEach((bundle, topicsMap) -> topicsMap.forEach((name, topic) -> {
- getTopicStats(topic, topicStats, includeConsumerMetrics, includeProducerMetrics,
- pulsar.getConfiguration().isExposePreciseBacklogInPrometheus(),
- pulsar.getConfiguration().isExposeSubscriptionBacklogSizeInPrometheus(),
- compactorMXBean
- );
+ try {
+ pulsar.getBrokerService().getMultiLayerTopicMap().forEach((namespace, bundlesMap) -> {
+ // only need to create if we are not including topic metrics
+ AggregatedNamespaceStats namespaceStats = includeTopicMetrics ? null : new AggregatedNamespaceStats();
+ topicsCount.reset();
+
+ bundlesMap.forEach((bundle, topicsMap) -> topicsMap.forEach((name, topic) -> {
+ //If not includeTopicMetrics then use a single thread local, so not assigning lots of objects
+ TopicStats topicStats = includeTopicMetrics ? new TopicStats() : localTopicStats.get();
+ topicStats.reset();
+ topicStats.name = name;
+ topicStats.namespace = namespace;
+ getTopicStats(topic, topicStats, includeConsumerMetrics, includeProducerMetrics,
+ pulsar.getConfiguration().isExposePreciseBacklogInPrometheus(),
+ pulsar.getConfiguration().isExposeSubscriptionBacklogSizeInPrometheus(),
+ compactorMXBean
+ );
+
+ if (includeTopicMetrics) {
+ topicsCount.add(1);
+ localAllTopicStats.get().add(topicStats);
+ } else {
+ namespaceStats.updateStats(topicStats);
+ }
+ }));
- if (includeTopicMetrics) {
- topicsCount.add(1);
- TopicStats.buildTopicStats(metrics, cluster, namespace, name, topicStats, compactorMXBean,
- splitTopicAndPartitionIndexLabel);
+ if (!includeTopicMetrics) {
+ // Only include namespace level stats if we don't have the per-topic, otherwise we're going to
+ // report the same data twice, and it will make the aggregation difficult
+ namespaceStats.name = namespace;
+ localAllNamespaceStats.get().add(namespaceStats);
} else {
- namespaceStats.updateStats(topicStats);
+ localNamespaceTopicCount.get().put(namespace, topicsCount.sum());
}
- }));
+ });
- if (!includeTopicMetrics) {
- // Only include namespace level stats if we don't have the per-topic, otherwise we're going to report
- // the same data twice, and it will make the aggregation difficult
- printNamespaceStats(metrics, cluster, namespace, namespaceStats);
+ if (includeTopicMetrics) {
+ printTopicsCountStats(stream, cluster, localNamespaceTopicCount.get());
+ TopicStats.printTopicStats(stream, cluster, localAllTopicStats.get(), compactorMXBean,
Review Comment:
Wouldn't you prefer keep the "printing" logic in `NamespaceAggregator` instead of spreading it around?
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/NamespaceStatsAggregator.java:
##########
@@ -305,155 +332,224 @@ private static void getTopicStats(Topic topic, TopicStats stats, boolean include
});
}
- private static void buildDefaultBrokerStats(Map<String, Collector.MetricFamilySamples> metrics, String cluster) {
- // Print metrics with 0 values. This is necessary to have the available brokers being
- // reported in the brokers dashboard even if they don't have any topic or traffic
- metric(metrics, cluster, "pulsar_topics_count", 0);
- metric(metrics, cluster, "pulsar_subscriptions_count", 0);
- metric(metrics, cluster, "pulsar_producers_count", 0);
- metric(metrics, cluster, "pulsar_consumers_count", 0);
- metric(metrics, cluster, "pulsar_rate_in", 0);
- metric(metrics, cluster, "pulsar_rate_out", 0);
- metric(metrics, cluster, "pulsar_throughput_in", 0);
- metric(metrics, cluster, "pulsar_throughput_out", 0);
- metric(metrics, cluster, "pulsar_storage_size", 0);
- metric(metrics, cluster, "pulsar_storage_logical_size", 0);
- metric(metrics, cluster, "pulsar_storage_write_rate", 0);
- metric(metrics, cluster, "pulsar_storage_read_rate", 0);
- metric(metrics, cluster, "pulsar_msg_backlog", 0);
+ private static void printTopicsCountStats(SimpleTextOutputStream stream, String cluster,
+ Map<String, Long> namespaceTopicsCount) {
+ stream.write("# TYPE ").write("pulsar_topics_count").write(" gauge\n");
+ stream.write("pulsar_topics_count")
+ .write("{cluster=\"").write(cluster).write("\"} ")
+ .write(0).write(' ').write(System.currentTimeMillis())
+ .write('\n');
+ namespaceTopicsCount.forEach((ns, topicCount) -> stream.write("pulsar_topics_count")
+ .write("{cluster=\"").write(cluster)
+ .write("\",namespace=\"").write(ns)
+ .write("\"} ")
+ .write(topicCount).write(' ').write(System.currentTimeMillis())
+ .write('\n')
+ );
}
- private static void printTopicsCountStats(Map<String, Collector.MetricFamilySamples> metrics, String cluster,
- String namespace,
- LongAdder topicsCount) {
- metric(metrics, cluster, namespace, "pulsar_topics_count", topicsCount.sum());
+ private static void printNamespaceStats(SimpleTextOutputStream stream, String cluster,
+ List<AggregatedNamespaceStats> stats) {
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_topics_count", stats, s -> s.topicsCount);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_subscriptions_count", stats, s -> s.subscriptionsCount);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_producers_count", stats, s -> s.producersCount);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_consumers_count", stats, s -> s.consumersCount);
+
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_rate_in", stats, s -> s.rateIn);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_rate_out", stats, s -> s.rateOut);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_throughput_in", stats, s -> s.throughputIn);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_throughput_out", stats, s -> s.throughputOut);
+ writeMetric(stream, cluster, "pulsar_consumer_msg_ack_rate", stats, s -> s.messageAckRate);
+
+ writeMetric(stream, cluster, "pulsar_in_bytes_total", stats, s -> s.bytesInCounter);
+ writeMetric(stream, cluster, "pulsar_in_messages_total", stats, s -> s.msgInCounter);
+ writeMetric(stream, cluster, "pulsar_out_bytes_total", stats, s -> s.bytesOutCounter);
+ writeMetric(stream, cluster, "pulsar_out_messages_total", stats, s -> s.msgOutCounter);
+
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_size", stats,
+ s -> s.managedLedgerStats.storageSize);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_logical_size", stats,
+ s -> s.managedLedgerStats.storageLogicalSize);
+ writeMetric(stream, cluster, "pulsar_storage_backlog_size", stats, s -> s.managedLedgerStats.backlogSize);
+ writeMetric(stream, cluster, "pulsar_storage_offloaded_size",
+ stats, s -> s.managedLedgerStats.offloadedStorageUsed);
+
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_write_rate", stats,
+ s -> s.managedLedgerStats.storageWriteRate);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_read_rate", stats,
+ s -> s.managedLedgerStats.storageReadRate);
+
+ writeMetric(stream, cluster, "pulsar_subscription_delayed", stats, s -> s.msgDelayed);
+
+ writeMsgBacklog(stream, cluster, stats, s -> s.msgBacklog);
+
+ stats.forEach(s -> s.managedLedgerStats.storageWriteLatencyBuckets.refresh());
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_0_5", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[0]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_1", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[1]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_5", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[2]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_10", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[3]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_20", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[4]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_50", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[5]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_100", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[6]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_200", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[7]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_1000", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[8]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_overflow", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[9]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_count",
+ stats, s -> s.managedLedgerStats.storageWriteLatencyBuckets.getCount());
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_sum",
+ stats, s -> s.managedLedgerStats.storageWriteLatencyBuckets.getSum());
+
+ stats.forEach(s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.refresh());
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_0_5", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[0]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_1", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[1]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_5", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[2]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_10", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[3]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_20", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[4]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_50", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[5]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_100", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[6]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_200", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[7]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_1000",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[8]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_overflow",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[9]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_count",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getCount());
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_sum",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getSum());
+
+ stats.forEach(s -> s.managedLedgerStats.entrySizeBuckets.refresh());
+ writeMetric(stream, cluster, "pulsar_entry_size_le_128", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[0]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_512", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[1]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_1_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[2]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_2_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[3]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_4_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[4]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_16_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[5]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_100_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[6]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_1_mb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[7]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_overflow", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[8]);
+ writeMetric(stream, cluster, "pulsar_entry_size_count",
+ stats, s -> s.managedLedgerStats.entrySizeBuckets.getCount());
+ writeMetric(stream, cluster, "pulsar_entry_size_sum",
+ stats, s -> s.managedLedgerStats.entrySizeBuckets.getSum());
+
+ writeReplicationStat(stream, cluster, "pulsar_replication_rate_in", stats,
+ replStats -> replStats.msgRateIn);
+ writeReplicationStat(stream, cluster, "pulsar_replication_rate_out", stats,
+ replStats -> replStats.msgRateOut);
+ writeReplicationStat(stream, cluster, "pulsar_replication_throughput_in", stats,
+ replStats -> replStats.msgThroughputIn);
+ writeReplicationStat(stream, cluster, "pulsar_replication_throughput_out", stats,
+ replStats -> replStats.msgThroughputOut);
+ writeReplicationStat(stream, cluster, "pulsar_replication_backlog", stats,
+ replStats -> replStats.replicationBacklog);
+ writeReplicationStat(stream, cluster, "pulsar_replication_connected_count", stats,
+ replStats -> replStats.connectedCount);
+ writeReplicationStat(stream, cluster, "pulsar_replication_rate_expired", stats,
+ replStats -> replStats.msgRateExpired);
+ writeReplicationStat(stream, cluster, "pulsar_replication_delay_in_seconds", stats,
+ replStats -> replStats.replicationDelayInSeconds);
}
- private static void printNamespaceStats(Map<String, Collector.MetricFamilySamples> metrics, String cluster,
- String namespace,
- AggregatedNamespaceStats stats) {
- metric(metrics, cluster, namespace, "pulsar_topics_count", stats.topicsCount);
- metric(metrics, cluster, namespace, "pulsar_subscriptions_count", stats.subscriptionsCount);
- metric(metrics, cluster, namespace, "pulsar_producers_count", stats.producersCount);
- metric(metrics, cluster, namespace, "pulsar_consumers_count", stats.consumersCount);
-
- metric(metrics, cluster, namespace, "pulsar_rate_in", stats.rateIn);
- metric(metrics, cluster, namespace, "pulsar_rate_out", stats.rateOut);
- metric(metrics, cluster, namespace, "pulsar_throughput_in", stats.throughputIn);
- metric(metrics, cluster, namespace, "pulsar_throughput_out", stats.throughputOut);
- metric(metrics, cluster, namespace, "pulsar_consumer_msg_ack_rate", stats.messageAckRate);
-
- metric(metrics, cluster, namespace, "pulsar_in_bytes_total", stats.bytesInCounter);
- metric(metrics, cluster, namespace, "pulsar_in_messages_total", stats.msgInCounter);
- metric(metrics, cluster, namespace, "pulsar_out_bytes_total", stats.bytesOutCounter);
- metric(metrics, cluster, namespace, "pulsar_out_messages_total", stats.msgOutCounter);
-
- metric(metrics, cluster, namespace, "pulsar_storage_size", stats.managedLedgerStats.storageSize);
- metric(metrics, cluster, namespace, "pulsar_storage_logical_size", stats.managedLedgerStats.storageLogicalSize);
- metric(metrics, cluster, namespace, "pulsar_storage_backlog_size", stats.managedLedgerStats.backlogSize);
- metric(metrics, cluster, namespace, "pulsar_storage_offloaded_size",
- stats.managedLedgerStats.offloadedStorageUsed);
-
- metric(metrics, cluster, namespace, "pulsar_storage_write_rate", stats.managedLedgerStats.storageWriteRate);
- metric(metrics, cluster, namespace, "pulsar_storage_read_rate", stats.managedLedgerStats.storageReadRate);
-
- metric(metrics, cluster, namespace, "pulsar_subscription_delayed", stats.msgDelayed);
-
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_msg_backlog", "local", stats.msgBacklog);
-
- stats.managedLedgerStats.storageWriteLatencyBuckets.refresh();
- long[] latencyBuckets = stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets();
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_0_5", latencyBuckets[0]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_1", latencyBuckets[1]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_5", latencyBuckets[2]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_10", latencyBuckets[3]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_20", latencyBuckets[4]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_50", latencyBuckets[5]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_100", latencyBuckets[6]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_200", latencyBuckets[7]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_1000", latencyBuckets[8]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_overflow", latencyBuckets[9]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_count",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getCount());
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_sum",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getSum());
-
- stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.refresh();
- long[] ledgerWritelatencyBuckets = stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets();
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_0_5", ledgerWritelatencyBuckets[0]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_1", ledgerWritelatencyBuckets[1]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_5", ledgerWritelatencyBuckets[2]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_10", ledgerWritelatencyBuckets[3]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_20", ledgerWritelatencyBuckets[4]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_50", ledgerWritelatencyBuckets[5]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_100", ledgerWritelatencyBuckets[6]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_200", ledgerWritelatencyBuckets[7]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_1000",
- ledgerWritelatencyBuckets[8]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_overflow",
- ledgerWritelatencyBuckets[9]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_count",
- stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getCount());
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_sum",
- stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getSum());
-
- stats.managedLedgerStats.entrySizeBuckets.refresh();
- long[] entrySizeBuckets = stats.managedLedgerStats.entrySizeBuckets.getBuckets();
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_128", entrySizeBuckets[0]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_512", entrySizeBuckets[1]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_1_kb", entrySizeBuckets[2]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_2_kb", entrySizeBuckets[3]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_4_kb", entrySizeBuckets[4]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_16_kb", entrySizeBuckets[5]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_100_kb", entrySizeBuckets[6]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_1_mb", entrySizeBuckets[7]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_overflow", entrySizeBuckets[8]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_count",
- stats.managedLedgerStats.entrySizeBuckets.getCount());
- metric(metrics, cluster, namespace, "pulsar_entry_size_sum",
- stats.managedLedgerStats.entrySizeBuckets.getSum());
-
- if (!stats.replicationStats.isEmpty()) {
- stats.replicationStats.forEach((remoteCluster, replStats) -> {
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_rate_in", remoteCluster,
- replStats.msgRateIn);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_rate_out", remoteCluster,
- replStats.msgRateOut);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_throughput_in", remoteCluster,
- replStats.msgThroughputIn);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_throughput_out", remoteCluster,
- replStats.msgThroughputOut);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_backlog", remoteCluster,
- replStats.replicationBacklog);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_connected_count",
- remoteCluster,
- replStats.connectedCount);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_rate_expired", remoteCluster,
- replStats.msgRateExpired);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_delay_in_seconds",
- remoteCluster, replStats.replicationDelayInSeconds);
- });
- }
+ private static void writeMetricWithBrokerDefault(SimpleTextOutputStream stream, String cluster, String name,
+ List<AggregatedNamespaceStats> allNamespaceStats,
+ Function<AggregatedNamespaceStats, Number> namespaceFunction) {
+ stream.write("# TYPE ").write(name).write(" gauge\n");
+ stream.write(name)
+ .write("{cluster=\"").write(cluster).write("\"} ")
+ .write(0).write(' ').write(System.currentTimeMillis())
+ .write('\n');
+ writeNamespaceStats(stream, cluster, name, allNamespaceStats, namespaceFunction);
}
- private static void metric(Map<String, Collector.MetricFamilySamples> metrics, String cluster, String name,
- long value) {
- addMetric(metrics, Map.of("cluster", cluster), name, value);
+ private static void writeMetric(SimpleTextOutputStream stream, String cluster, String name,
+ List<AggregatedNamespaceStats> allNamespaceStats,
+ Function<AggregatedNamespaceStats, Number> namespaceFunction) {
+ stream.write("# TYPE ").write(name).write(" gauge\n");
+ writeNamespaceStats(stream, cluster, name, allNamespaceStats, namespaceFunction);
}
- private static void metric(Map<String, Collector.MetricFamilySamples> metrics, String cluster, String namespace,
- String name, long value) {
- addMetric(metrics, Map.of("cluster", cluster, "namespace", namespace), name, value);
+ private static void writeNamespaceStats(SimpleTextOutputStream stream, String cluster, String name,
+ List<AggregatedNamespaceStats> allNamespaceStats,
+ Function<AggregatedNamespaceStats, Number> namespaceFunction) {
Review Comment:
`namespaceFunction` --> `valueExtractFunction`
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/NamespaceStatsAggregator.java:
##########
@@ -305,155 +332,224 @@ private static void getTopicStats(Topic topic, TopicStats stats, boolean include
});
}
- private static void buildDefaultBrokerStats(Map<String, Collector.MetricFamilySamples> metrics, String cluster) {
- // Print metrics with 0 values. This is necessary to have the available brokers being
- // reported in the brokers dashboard even if they don't have any topic or traffic
- metric(metrics, cluster, "pulsar_topics_count", 0);
- metric(metrics, cluster, "pulsar_subscriptions_count", 0);
- metric(metrics, cluster, "pulsar_producers_count", 0);
- metric(metrics, cluster, "pulsar_consumers_count", 0);
- metric(metrics, cluster, "pulsar_rate_in", 0);
- metric(metrics, cluster, "pulsar_rate_out", 0);
- metric(metrics, cluster, "pulsar_throughput_in", 0);
- metric(metrics, cluster, "pulsar_throughput_out", 0);
- metric(metrics, cluster, "pulsar_storage_size", 0);
- metric(metrics, cluster, "pulsar_storage_logical_size", 0);
- metric(metrics, cluster, "pulsar_storage_write_rate", 0);
- metric(metrics, cluster, "pulsar_storage_read_rate", 0);
- metric(metrics, cluster, "pulsar_msg_backlog", 0);
+ private static void printTopicsCountStats(SimpleTextOutputStream stream, String cluster,
+ Map<String, Long> namespaceTopicsCount) {
+ stream.write("# TYPE ").write("pulsar_topics_count").write(" gauge\n");
+ stream.write("pulsar_topics_count")
+ .write("{cluster=\"").write(cluster).write("\"} ")
+ .write(0).write(' ').write(System.currentTimeMillis())
+ .write('\n');
+ namespaceTopicsCount.forEach((ns, topicCount) -> stream.write("pulsar_topics_count")
+ .write("{cluster=\"").write(cluster)
+ .write("\",namespace=\"").write(ns)
+ .write("\"} ")
+ .write(topicCount).write(' ').write(System.currentTimeMillis())
+ .write('\n')
+ );
}
- private static void printTopicsCountStats(Map<String, Collector.MetricFamilySamples> metrics, String cluster,
- String namespace,
- LongAdder topicsCount) {
- metric(metrics, cluster, namespace, "pulsar_topics_count", topicsCount.sum());
+ private static void printNamespaceStats(SimpleTextOutputStream stream, String cluster,
+ List<AggregatedNamespaceStats> stats) {
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_topics_count", stats, s -> s.topicsCount);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_subscriptions_count", stats, s -> s.subscriptionsCount);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_producers_count", stats, s -> s.producersCount);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_consumers_count", stats, s -> s.consumersCount);
+
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_rate_in", stats, s -> s.rateIn);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_rate_out", stats, s -> s.rateOut);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_throughput_in", stats, s -> s.throughputIn);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_throughput_out", stats, s -> s.throughputOut);
+ writeMetric(stream, cluster, "pulsar_consumer_msg_ack_rate", stats, s -> s.messageAckRate);
+
+ writeMetric(stream, cluster, "pulsar_in_bytes_total", stats, s -> s.bytesInCounter);
+ writeMetric(stream, cluster, "pulsar_in_messages_total", stats, s -> s.msgInCounter);
+ writeMetric(stream, cluster, "pulsar_out_bytes_total", stats, s -> s.bytesOutCounter);
+ writeMetric(stream, cluster, "pulsar_out_messages_total", stats, s -> s.msgOutCounter);
+
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_size", stats,
+ s -> s.managedLedgerStats.storageSize);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_logical_size", stats,
+ s -> s.managedLedgerStats.storageLogicalSize);
+ writeMetric(stream, cluster, "pulsar_storage_backlog_size", stats, s -> s.managedLedgerStats.backlogSize);
+ writeMetric(stream, cluster, "pulsar_storage_offloaded_size",
+ stats, s -> s.managedLedgerStats.offloadedStorageUsed);
+
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_write_rate", stats,
+ s -> s.managedLedgerStats.storageWriteRate);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_read_rate", stats,
+ s -> s.managedLedgerStats.storageReadRate);
+
+ writeMetric(stream, cluster, "pulsar_subscription_delayed", stats, s -> s.msgDelayed);
+
+ writeMsgBacklog(stream, cluster, stats, s -> s.msgBacklog);
+
+ stats.forEach(s -> s.managedLedgerStats.storageWriteLatencyBuckets.refresh());
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_0_5", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[0]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_1", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[1]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_5", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[2]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_10", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[3]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_20", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[4]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_50", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[5]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_100", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[6]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_200", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[7]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_1000", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[8]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_overflow", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[9]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_count",
+ stats, s -> s.managedLedgerStats.storageWriteLatencyBuckets.getCount());
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_sum",
+ stats, s -> s.managedLedgerStats.storageWriteLatencyBuckets.getSum());
+
+ stats.forEach(s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.refresh());
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_0_5", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[0]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_1", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[1]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_5", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[2]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_10", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[3]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_20", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[4]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_50", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[5]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_100", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[6]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_200", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[7]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_1000",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[8]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_overflow",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[9]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_count",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getCount());
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_sum",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getSum());
+
+ stats.forEach(s -> s.managedLedgerStats.entrySizeBuckets.refresh());
+ writeMetric(stream, cluster, "pulsar_entry_size_le_128", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[0]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_512", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[1]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_1_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[2]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_2_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[3]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_4_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[4]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_16_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[5]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_100_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[6]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_1_mb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[7]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_overflow", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[8]);
+ writeMetric(stream, cluster, "pulsar_entry_size_count",
+ stats, s -> s.managedLedgerStats.entrySizeBuckets.getCount());
+ writeMetric(stream, cluster, "pulsar_entry_size_sum",
+ stats, s -> s.managedLedgerStats.entrySizeBuckets.getSum());
+
+ writeReplicationStat(stream, cluster, "pulsar_replication_rate_in", stats,
+ replStats -> replStats.msgRateIn);
+ writeReplicationStat(stream, cluster, "pulsar_replication_rate_out", stats,
+ replStats -> replStats.msgRateOut);
+ writeReplicationStat(stream, cluster, "pulsar_replication_throughput_in", stats,
+ replStats -> replStats.msgThroughputIn);
+ writeReplicationStat(stream, cluster, "pulsar_replication_throughput_out", stats,
+ replStats -> replStats.msgThroughputOut);
+ writeReplicationStat(stream, cluster, "pulsar_replication_backlog", stats,
+ replStats -> replStats.replicationBacklog);
+ writeReplicationStat(stream, cluster, "pulsar_replication_connected_count", stats,
+ replStats -> replStats.connectedCount);
+ writeReplicationStat(stream, cluster, "pulsar_replication_rate_expired", stats,
+ replStats -> replStats.msgRateExpired);
+ writeReplicationStat(stream, cluster, "pulsar_replication_delay_in_seconds", stats,
+ replStats -> replStats.replicationDelayInSeconds);
}
- private static void printNamespaceStats(Map<String, Collector.MetricFamilySamples> metrics, String cluster,
- String namespace,
- AggregatedNamespaceStats stats) {
- metric(metrics, cluster, namespace, "pulsar_topics_count", stats.topicsCount);
- metric(metrics, cluster, namespace, "pulsar_subscriptions_count", stats.subscriptionsCount);
- metric(metrics, cluster, namespace, "pulsar_producers_count", stats.producersCount);
- metric(metrics, cluster, namespace, "pulsar_consumers_count", stats.consumersCount);
-
- metric(metrics, cluster, namespace, "pulsar_rate_in", stats.rateIn);
- metric(metrics, cluster, namespace, "pulsar_rate_out", stats.rateOut);
- metric(metrics, cluster, namespace, "pulsar_throughput_in", stats.throughputIn);
- metric(metrics, cluster, namespace, "pulsar_throughput_out", stats.throughputOut);
- metric(metrics, cluster, namespace, "pulsar_consumer_msg_ack_rate", stats.messageAckRate);
-
- metric(metrics, cluster, namespace, "pulsar_in_bytes_total", stats.bytesInCounter);
- metric(metrics, cluster, namespace, "pulsar_in_messages_total", stats.msgInCounter);
- metric(metrics, cluster, namespace, "pulsar_out_bytes_total", stats.bytesOutCounter);
- metric(metrics, cluster, namespace, "pulsar_out_messages_total", stats.msgOutCounter);
-
- metric(metrics, cluster, namespace, "pulsar_storage_size", stats.managedLedgerStats.storageSize);
- metric(metrics, cluster, namespace, "pulsar_storage_logical_size", stats.managedLedgerStats.storageLogicalSize);
- metric(metrics, cluster, namespace, "pulsar_storage_backlog_size", stats.managedLedgerStats.backlogSize);
- metric(metrics, cluster, namespace, "pulsar_storage_offloaded_size",
- stats.managedLedgerStats.offloadedStorageUsed);
-
- metric(metrics, cluster, namespace, "pulsar_storage_write_rate", stats.managedLedgerStats.storageWriteRate);
- metric(metrics, cluster, namespace, "pulsar_storage_read_rate", stats.managedLedgerStats.storageReadRate);
-
- metric(metrics, cluster, namespace, "pulsar_subscription_delayed", stats.msgDelayed);
-
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_msg_backlog", "local", stats.msgBacklog);
-
- stats.managedLedgerStats.storageWriteLatencyBuckets.refresh();
- long[] latencyBuckets = stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets();
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_0_5", latencyBuckets[0]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_1", latencyBuckets[1]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_5", latencyBuckets[2]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_10", latencyBuckets[3]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_20", latencyBuckets[4]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_50", latencyBuckets[5]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_100", latencyBuckets[6]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_200", latencyBuckets[7]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_1000", latencyBuckets[8]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_overflow", latencyBuckets[9]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_count",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getCount());
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_sum",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getSum());
-
- stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.refresh();
- long[] ledgerWritelatencyBuckets = stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets();
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_0_5", ledgerWritelatencyBuckets[0]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_1", ledgerWritelatencyBuckets[1]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_5", ledgerWritelatencyBuckets[2]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_10", ledgerWritelatencyBuckets[3]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_20", ledgerWritelatencyBuckets[4]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_50", ledgerWritelatencyBuckets[5]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_100", ledgerWritelatencyBuckets[6]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_200", ledgerWritelatencyBuckets[7]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_1000",
- ledgerWritelatencyBuckets[8]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_overflow",
- ledgerWritelatencyBuckets[9]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_count",
- stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getCount());
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_sum",
- stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getSum());
-
- stats.managedLedgerStats.entrySizeBuckets.refresh();
- long[] entrySizeBuckets = stats.managedLedgerStats.entrySizeBuckets.getBuckets();
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_128", entrySizeBuckets[0]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_512", entrySizeBuckets[1]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_1_kb", entrySizeBuckets[2]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_2_kb", entrySizeBuckets[3]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_4_kb", entrySizeBuckets[4]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_16_kb", entrySizeBuckets[5]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_100_kb", entrySizeBuckets[6]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_1_mb", entrySizeBuckets[7]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_overflow", entrySizeBuckets[8]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_count",
- stats.managedLedgerStats.entrySizeBuckets.getCount());
- metric(metrics, cluster, namespace, "pulsar_entry_size_sum",
- stats.managedLedgerStats.entrySizeBuckets.getSum());
-
- if (!stats.replicationStats.isEmpty()) {
- stats.replicationStats.forEach((remoteCluster, replStats) -> {
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_rate_in", remoteCluster,
- replStats.msgRateIn);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_rate_out", remoteCluster,
- replStats.msgRateOut);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_throughput_in", remoteCluster,
- replStats.msgThroughputIn);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_throughput_out", remoteCluster,
- replStats.msgThroughputOut);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_backlog", remoteCluster,
- replStats.replicationBacklog);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_connected_count",
- remoteCluster,
- replStats.connectedCount);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_rate_expired", remoteCluster,
- replStats.msgRateExpired);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_delay_in_seconds",
- remoteCluster, replStats.replicationDelayInSeconds);
- });
- }
+ private static void writeMetricWithBrokerDefault(SimpleTextOutputStream stream, String cluster, String name,
+ List<AggregatedNamespaceStats> allNamespaceStats,
+ Function<AggregatedNamespaceStats, Number> namespaceFunction) {
+ stream.write("# TYPE ").write(name).write(" gauge\n");
+ stream.write(name)
+ .write("{cluster=\"").write(cluster).write("\"} ")
+ .write(0).write(' ').write(System.currentTimeMillis())
+ .write('\n');
+ writeNamespaceStats(stream, cluster, name, allNamespaceStats, namespaceFunction);
}
- private static void metric(Map<String, Collector.MetricFamilySamples> metrics, String cluster, String name,
- long value) {
- addMetric(metrics, Map.of("cluster", cluster), name, value);
+ private static void writeMetric(SimpleTextOutputStream stream, String cluster, String name,
+ List<AggregatedNamespaceStats> allNamespaceStats,
+ Function<AggregatedNamespaceStats, Number> namespaceFunction) {
+ stream.write("# TYPE ").write(name).write(" gauge\n");
+ writeNamespaceStats(stream, cluster, name, allNamespaceStats, namespaceFunction);
}
- private static void metric(Map<String, Collector.MetricFamilySamples> metrics, String cluster, String namespace,
- String name, long value) {
- addMetric(metrics, Map.of("cluster", cluster, "namespace", namespace), name, value);
+ private static void writeNamespaceStats(SimpleTextOutputStream stream, String cluster, String name,
Review Comment:
`name` --> `metricName`
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/NamespaceStatsAggregator.java:
##########
@@ -305,155 +332,224 @@ private static void getTopicStats(Topic topic, TopicStats stats, boolean include
});
}
- private static void buildDefaultBrokerStats(Map<String, Collector.MetricFamilySamples> metrics, String cluster) {
- // Print metrics with 0 values. This is necessary to have the available brokers being
- // reported in the brokers dashboard even if they don't have any topic or traffic
- metric(metrics, cluster, "pulsar_topics_count", 0);
- metric(metrics, cluster, "pulsar_subscriptions_count", 0);
- metric(metrics, cluster, "pulsar_producers_count", 0);
- metric(metrics, cluster, "pulsar_consumers_count", 0);
- metric(metrics, cluster, "pulsar_rate_in", 0);
- metric(metrics, cluster, "pulsar_rate_out", 0);
- metric(metrics, cluster, "pulsar_throughput_in", 0);
- metric(metrics, cluster, "pulsar_throughput_out", 0);
- metric(metrics, cluster, "pulsar_storage_size", 0);
- metric(metrics, cluster, "pulsar_storage_logical_size", 0);
- metric(metrics, cluster, "pulsar_storage_write_rate", 0);
- metric(metrics, cluster, "pulsar_storage_read_rate", 0);
- metric(metrics, cluster, "pulsar_msg_backlog", 0);
+ private static void printTopicsCountStats(SimpleTextOutputStream stream, String cluster,
+ Map<String, Long> namespaceTopicsCount) {
+ stream.write("# TYPE ").write("pulsar_topics_count").write(" gauge\n");
+ stream.write("pulsar_topics_count")
+ .write("{cluster=\"").write(cluster).write("\"} ")
+ .write(0).write(' ').write(System.currentTimeMillis())
+ .write('\n');
+ namespaceTopicsCount.forEach((ns, topicCount) -> stream.write("pulsar_topics_count")
+ .write("{cluster=\"").write(cluster)
+ .write("\",namespace=\"").write(ns)
+ .write("\"} ")
+ .write(topicCount).write(' ').write(System.currentTimeMillis())
+ .write('\n')
+ );
}
- private static void printTopicsCountStats(Map<String, Collector.MetricFamilySamples> metrics, String cluster,
- String namespace,
- LongAdder topicsCount) {
- metric(metrics, cluster, namespace, "pulsar_topics_count", topicsCount.sum());
+ private static void printNamespaceStats(SimpleTextOutputStream stream, String cluster,
+ List<AggregatedNamespaceStats> stats) {
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_topics_count", stats, s -> s.topicsCount);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_subscriptions_count", stats, s -> s.subscriptionsCount);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_producers_count", stats, s -> s.producersCount);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_consumers_count", stats, s -> s.consumersCount);
+
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_rate_in", stats, s -> s.rateIn);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_rate_out", stats, s -> s.rateOut);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_throughput_in", stats, s -> s.throughputIn);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_throughput_out", stats, s -> s.throughputOut);
+ writeMetric(stream, cluster, "pulsar_consumer_msg_ack_rate", stats, s -> s.messageAckRate);
+
+ writeMetric(stream, cluster, "pulsar_in_bytes_total", stats, s -> s.bytesInCounter);
+ writeMetric(stream, cluster, "pulsar_in_messages_total", stats, s -> s.msgInCounter);
+ writeMetric(stream, cluster, "pulsar_out_bytes_total", stats, s -> s.bytesOutCounter);
+ writeMetric(stream, cluster, "pulsar_out_messages_total", stats, s -> s.msgOutCounter);
+
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_size", stats,
+ s -> s.managedLedgerStats.storageSize);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_logical_size", stats,
+ s -> s.managedLedgerStats.storageLogicalSize);
+ writeMetric(stream, cluster, "pulsar_storage_backlog_size", stats, s -> s.managedLedgerStats.backlogSize);
+ writeMetric(stream, cluster, "pulsar_storage_offloaded_size",
+ stats, s -> s.managedLedgerStats.offloadedStorageUsed);
+
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_write_rate", stats,
+ s -> s.managedLedgerStats.storageWriteRate);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_read_rate", stats,
+ s -> s.managedLedgerStats.storageReadRate);
+
+ writeMetric(stream, cluster, "pulsar_subscription_delayed", stats, s -> s.msgDelayed);
+
+ writeMsgBacklog(stream, cluster, stats, s -> s.msgBacklog);
+
+ stats.forEach(s -> s.managedLedgerStats.storageWriteLatencyBuckets.refresh());
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_0_5", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[0]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_1", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[1]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_5", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[2]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_10", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[3]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_20", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[4]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_50", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[5]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_100", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[6]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_200", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[7]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_1000", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[8]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_overflow", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[9]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_count",
+ stats, s -> s.managedLedgerStats.storageWriteLatencyBuckets.getCount());
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_sum",
+ stats, s -> s.managedLedgerStats.storageWriteLatencyBuckets.getSum());
+
+ stats.forEach(s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.refresh());
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_0_5", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[0]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_1", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[1]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_5", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[2]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_10", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[3]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_20", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[4]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_50", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[5]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_100", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[6]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_200", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[7]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_1000",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[8]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_overflow",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[9]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_count",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getCount());
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_sum",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getSum());
+
+ stats.forEach(s -> s.managedLedgerStats.entrySizeBuckets.refresh());
+ writeMetric(stream, cluster, "pulsar_entry_size_le_128", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[0]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_512", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[1]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_1_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[2]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_2_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[3]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_4_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[4]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_16_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[5]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_100_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[6]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_1_mb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[7]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_overflow", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[8]);
+ writeMetric(stream, cluster, "pulsar_entry_size_count",
+ stats, s -> s.managedLedgerStats.entrySizeBuckets.getCount());
+ writeMetric(stream, cluster, "pulsar_entry_size_sum",
+ stats, s -> s.managedLedgerStats.entrySizeBuckets.getSum());
+
+ writeReplicationStat(stream, cluster, "pulsar_replication_rate_in", stats,
+ replStats -> replStats.msgRateIn);
+ writeReplicationStat(stream, cluster, "pulsar_replication_rate_out", stats,
+ replStats -> replStats.msgRateOut);
+ writeReplicationStat(stream, cluster, "pulsar_replication_throughput_in", stats,
+ replStats -> replStats.msgThroughputIn);
+ writeReplicationStat(stream, cluster, "pulsar_replication_throughput_out", stats,
+ replStats -> replStats.msgThroughputOut);
+ writeReplicationStat(stream, cluster, "pulsar_replication_backlog", stats,
+ replStats -> replStats.replicationBacklog);
+ writeReplicationStat(stream, cluster, "pulsar_replication_connected_count", stats,
+ replStats -> replStats.connectedCount);
+ writeReplicationStat(stream, cluster, "pulsar_replication_rate_expired", stats,
+ replStats -> replStats.msgRateExpired);
+ writeReplicationStat(stream, cluster, "pulsar_replication_delay_in_seconds", stats,
+ replStats -> replStats.replicationDelayInSeconds);
}
- private static void printNamespaceStats(Map<String, Collector.MetricFamilySamples> metrics, String cluster,
- String namespace,
- AggregatedNamespaceStats stats) {
- metric(metrics, cluster, namespace, "pulsar_topics_count", stats.topicsCount);
- metric(metrics, cluster, namespace, "pulsar_subscriptions_count", stats.subscriptionsCount);
- metric(metrics, cluster, namespace, "pulsar_producers_count", stats.producersCount);
- metric(metrics, cluster, namespace, "pulsar_consumers_count", stats.consumersCount);
-
- metric(metrics, cluster, namespace, "pulsar_rate_in", stats.rateIn);
- metric(metrics, cluster, namespace, "pulsar_rate_out", stats.rateOut);
- metric(metrics, cluster, namespace, "pulsar_throughput_in", stats.throughputIn);
- metric(metrics, cluster, namespace, "pulsar_throughput_out", stats.throughputOut);
- metric(metrics, cluster, namespace, "pulsar_consumer_msg_ack_rate", stats.messageAckRate);
-
- metric(metrics, cluster, namespace, "pulsar_in_bytes_total", stats.bytesInCounter);
- metric(metrics, cluster, namespace, "pulsar_in_messages_total", stats.msgInCounter);
- metric(metrics, cluster, namespace, "pulsar_out_bytes_total", stats.bytesOutCounter);
- metric(metrics, cluster, namespace, "pulsar_out_messages_total", stats.msgOutCounter);
-
- metric(metrics, cluster, namespace, "pulsar_storage_size", stats.managedLedgerStats.storageSize);
- metric(metrics, cluster, namespace, "pulsar_storage_logical_size", stats.managedLedgerStats.storageLogicalSize);
- metric(metrics, cluster, namespace, "pulsar_storage_backlog_size", stats.managedLedgerStats.backlogSize);
- metric(metrics, cluster, namespace, "pulsar_storage_offloaded_size",
- stats.managedLedgerStats.offloadedStorageUsed);
-
- metric(metrics, cluster, namespace, "pulsar_storage_write_rate", stats.managedLedgerStats.storageWriteRate);
- metric(metrics, cluster, namespace, "pulsar_storage_read_rate", stats.managedLedgerStats.storageReadRate);
-
- metric(metrics, cluster, namespace, "pulsar_subscription_delayed", stats.msgDelayed);
-
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_msg_backlog", "local", stats.msgBacklog);
-
- stats.managedLedgerStats.storageWriteLatencyBuckets.refresh();
- long[] latencyBuckets = stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets();
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_0_5", latencyBuckets[0]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_1", latencyBuckets[1]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_5", latencyBuckets[2]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_10", latencyBuckets[3]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_20", latencyBuckets[4]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_50", latencyBuckets[5]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_100", latencyBuckets[6]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_200", latencyBuckets[7]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_1000", latencyBuckets[8]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_overflow", latencyBuckets[9]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_count",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getCount());
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_sum",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getSum());
-
- stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.refresh();
- long[] ledgerWritelatencyBuckets = stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets();
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_0_5", ledgerWritelatencyBuckets[0]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_1", ledgerWritelatencyBuckets[1]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_5", ledgerWritelatencyBuckets[2]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_10", ledgerWritelatencyBuckets[3]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_20", ledgerWritelatencyBuckets[4]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_50", ledgerWritelatencyBuckets[5]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_100", ledgerWritelatencyBuckets[6]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_200", ledgerWritelatencyBuckets[7]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_1000",
- ledgerWritelatencyBuckets[8]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_overflow",
- ledgerWritelatencyBuckets[9]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_count",
- stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getCount());
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_sum",
- stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getSum());
-
- stats.managedLedgerStats.entrySizeBuckets.refresh();
- long[] entrySizeBuckets = stats.managedLedgerStats.entrySizeBuckets.getBuckets();
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_128", entrySizeBuckets[0]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_512", entrySizeBuckets[1]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_1_kb", entrySizeBuckets[2]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_2_kb", entrySizeBuckets[3]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_4_kb", entrySizeBuckets[4]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_16_kb", entrySizeBuckets[5]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_100_kb", entrySizeBuckets[6]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_1_mb", entrySizeBuckets[7]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_overflow", entrySizeBuckets[8]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_count",
- stats.managedLedgerStats.entrySizeBuckets.getCount());
- metric(metrics, cluster, namespace, "pulsar_entry_size_sum",
- stats.managedLedgerStats.entrySizeBuckets.getSum());
-
- if (!stats.replicationStats.isEmpty()) {
- stats.replicationStats.forEach((remoteCluster, replStats) -> {
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_rate_in", remoteCluster,
- replStats.msgRateIn);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_rate_out", remoteCluster,
- replStats.msgRateOut);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_throughput_in", remoteCluster,
- replStats.msgThroughputIn);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_throughput_out", remoteCluster,
- replStats.msgThroughputOut);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_backlog", remoteCluster,
- replStats.replicationBacklog);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_connected_count",
- remoteCluster,
- replStats.connectedCount);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_rate_expired", remoteCluster,
- replStats.msgRateExpired);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_delay_in_seconds",
- remoteCluster, replStats.replicationDelayInSeconds);
- });
- }
+ private static void writeMetricWithBrokerDefault(SimpleTextOutputStream stream, String cluster, String name,
+ List<AggregatedNamespaceStats> allNamespaceStats,
+ Function<AggregatedNamespaceStats, Number> namespaceFunction) {
+ stream.write("# TYPE ").write(name).write(" gauge\n");
+ stream.write(name)
Review Comment:
Let's condition it.
```
if (allNamespaceStats.isEmpty) {
// write the default line at cluster level
} else {
writeNamespaceStats(stream, cluster, name, allNamespaceStats, namespaceFunction);
{
You refactor makes more sense, and if you condition it, it will remove writing the same metric twice.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/NamespaceStatsAggregator.java:
##########
@@ -305,155 +332,224 @@ private static void getTopicStats(Topic topic, TopicStats stats, boolean include
});
}
- private static void buildDefaultBrokerStats(Map<String, Collector.MetricFamilySamples> metrics, String cluster) {
- // Print metrics with 0 values. This is necessary to have the available brokers being
- // reported in the brokers dashboard even if they don't have any topic or traffic
- metric(metrics, cluster, "pulsar_topics_count", 0);
- metric(metrics, cluster, "pulsar_subscriptions_count", 0);
- metric(metrics, cluster, "pulsar_producers_count", 0);
- metric(metrics, cluster, "pulsar_consumers_count", 0);
- metric(metrics, cluster, "pulsar_rate_in", 0);
- metric(metrics, cluster, "pulsar_rate_out", 0);
- metric(metrics, cluster, "pulsar_throughput_in", 0);
- metric(metrics, cluster, "pulsar_throughput_out", 0);
- metric(metrics, cluster, "pulsar_storage_size", 0);
- metric(metrics, cluster, "pulsar_storage_logical_size", 0);
- metric(metrics, cluster, "pulsar_storage_write_rate", 0);
- metric(metrics, cluster, "pulsar_storage_read_rate", 0);
- metric(metrics, cluster, "pulsar_msg_backlog", 0);
+ private static void printTopicsCountStats(SimpleTextOutputStream stream, String cluster,
+ Map<String, Long> namespaceTopicsCount) {
+ stream.write("# TYPE ").write("pulsar_topics_count").write(" gauge\n");
+ stream.write("pulsar_topics_count")
+ .write("{cluster=\"").write(cluster).write("\"} ")
+ .write(0).write(' ').write(System.currentTimeMillis())
+ .write('\n');
+ namespaceTopicsCount.forEach((ns, topicCount) -> stream.write("pulsar_topics_count")
+ .write("{cluster=\"").write(cluster)
+ .write("\",namespace=\"").write(ns)
+ .write("\"} ")
+ .write(topicCount).write(' ').write(System.currentTimeMillis())
+ .write('\n')
+ );
}
- private static void printTopicsCountStats(Map<String, Collector.MetricFamilySamples> metrics, String cluster,
- String namespace,
- LongAdder topicsCount) {
- metric(metrics, cluster, namespace, "pulsar_topics_count", topicsCount.sum());
+ private static void printNamespaceStats(SimpleTextOutputStream stream, String cluster,
+ List<AggregatedNamespaceStats> stats) {
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_topics_count", stats, s -> s.topicsCount);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_subscriptions_count", stats, s -> s.subscriptionsCount);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_producers_count", stats, s -> s.producersCount);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_consumers_count", stats, s -> s.consumersCount);
+
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_rate_in", stats, s -> s.rateIn);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_rate_out", stats, s -> s.rateOut);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_throughput_in", stats, s -> s.throughputIn);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_throughput_out", stats, s -> s.throughputOut);
+ writeMetric(stream, cluster, "pulsar_consumer_msg_ack_rate", stats, s -> s.messageAckRate);
+
+ writeMetric(stream, cluster, "pulsar_in_bytes_total", stats, s -> s.bytesInCounter);
+ writeMetric(stream, cluster, "pulsar_in_messages_total", stats, s -> s.msgInCounter);
+ writeMetric(stream, cluster, "pulsar_out_bytes_total", stats, s -> s.bytesOutCounter);
+ writeMetric(stream, cluster, "pulsar_out_messages_total", stats, s -> s.msgOutCounter);
+
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_size", stats,
+ s -> s.managedLedgerStats.storageSize);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_logical_size", stats,
+ s -> s.managedLedgerStats.storageLogicalSize);
+ writeMetric(stream, cluster, "pulsar_storage_backlog_size", stats, s -> s.managedLedgerStats.backlogSize);
+ writeMetric(stream, cluster, "pulsar_storage_offloaded_size",
+ stats, s -> s.managedLedgerStats.offloadedStorageUsed);
+
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_write_rate", stats,
+ s -> s.managedLedgerStats.storageWriteRate);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_read_rate", stats,
+ s -> s.managedLedgerStats.storageReadRate);
+
+ writeMetric(stream, cluster, "pulsar_subscription_delayed", stats, s -> s.msgDelayed);
+
+ writeMsgBacklog(stream, cluster, stats, s -> s.msgBacklog);
+
+ stats.forEach(s -> s.managedLedgerStats.storageWriteLatencyBuckets.refresh());
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_0_5", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[0]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_1", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[1]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_5", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[2]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_10", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[3]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_20", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[4]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_50", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[5]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_100", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[6]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_200", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[7]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_1000", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[8]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_overflow", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[9]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_count",
+ stats, s -> s.managedLedgerStats.storageWriteLatencyBuckets.getCount());
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_sum",
+ stats, s -> s.managedLedgerStats.storageWriteLatencyBuckets.getSum());
+
+ stats.forEach(s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.refresh());
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_0_5", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[0]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_1", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[1]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_5", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[2]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_10", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[3]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_20", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[4]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_50", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[5]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_100", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[6]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_200", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[7]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_1000",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[8]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_overflow",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[9]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_count",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getCount());
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_sum",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getSum());
+
+ stats.forEach(s -> s.managedLedgerStats.entrySizeBuckets.refresh());
+ writeMetric(stream, cluster, "pulsar_entry_size_le_128", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[0]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_512", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[1]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_1_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[2]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_2_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[3]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_4_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[4]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_16_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[5]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_100_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[6]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_1_mb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[7]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_overflow", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[8]);
+ writeMetric(stream, cluster, "pulsar_entry_size_count",
+ stats, s -> s.managedLedgerStats.entrySizeBuckets.getCount());
+ writeMetric(stream, cluster, "pulsar_entry_size_sum",
+ stats, s -> s.managedLedgerStats.entrySizeBuckets.getSum());
+
+ writeReplicationStat(stream, cluster, "pulsar_replication_rate_in", stats,
+ replStats -> replStats.msgRateIn);
+ writeReplicationStat(stream, cluster, "pulsar_replication_rate_out", stats,
+ replStats -> replStats.msgRateOut);
+ writeReplicationStat(stream, cluster, "pulsar_replication_throughput_in", stats,
+ replStats -> replStats.msgThroughputIn);
+ writeReplicationStat(stream, cluster, "pulsar_replication_throughput_out", stats,
+ replStats -> replStats.msgThroughputOut);
+ writeReplicationStat(stream, cluster, "pulsar_replication_backlog", stats,
+ replStats -> replStats.replicationBacklog);
+ writeReplicationStat(stream, cluster, "pulsar_replication_connected_count", stats,
+ replStats -> replStats.connectedCount);
+ writeReplicationStat(stream, cluster, "pulsar_replication_rate_expired", stats,
+ replStats -> replStats.msgRateExpired);
+ writeReplicationStat(stream, cluster, "pulsar_replication_delay_in_seconds", stats,
+ replStats -> replStats.replicationDelayInSeconds);
}
- private static void printNamespaceStats(Map<String, Collector.MetricFamilySamples> metrics, String cluster,
- String namespace,
- AggregatedNamespaceStats stats) {
- metric(metrics, cluster, namespace, "pulsar_topics_count", stats.topicsCount);
- metric(metrics, cluster, namespace, "pulsar_subscriptions_count", stats.subscriptionsCount);
- metric(metrics, cluster, namespace, "pulsar_producers_count", stats.producersCount);
- metric(metrics, cluster, namespace, "pulsar_consumers_count", stats.consumersCount);
-
- metric(metrics, cluster, namespace, "pulsar_rate_in", stats.rateIn);
- metric(metrics, cluster, namespace, "pulsar_rate_out", stats.rateOut);
- metric(metrics, cluster, namespace, "pulsar_throughput_in", stats.throughputIn);
- metric(metrics, cluster, namespace, "pulsar_throughput_out", stats.throughputOut);
- metric(metrics, cluster, namespace, "pulsar_consumer_msg_ack_rate", stats.messageAckRate);
-
- metric(metrics, cluster, namespace, "pulsar_in_bytes_total", stats.bytesInCounter);
- metric(metrics, cluster, namespace, "pulsar_in_messages_total", stats.msgInCounter);
- metric(metrics, cluster, namespace, "pulsar_out_bytes_total", stats.bytesOutCounter);
- metric(metrics, cluster, namespace, "pulsar_out_messages_total", stats.msgOutCounter);
-
- metric(metrics, cluster, namespace, "pulsar_storage_size", stats.managedLedgerStats.storageSize);
- metric(metrics, cluster, namespace, "pulsar_storage_logical_size", stats.managedLedgerStats.storageLogicalSize);
- metric(metrics, cluster, namespace, "pulsar_storage_backlog_size", stats.managedLedgerStats.backlogSize);
- metric(metrics, cluster, namespace, "pulsar_storage_offloaded_size",
- stats.managedLedgerStats.offloadedStorageUsed);
-
- metric(metrics, cluster, namespace, "pulsar_storage_write_rate", stats.managedLedgerStats.storageWriteRate);
- metric(metrics, cluster, namespace, "pulsar_storage_read_rate", stats.managedLedgerStats.storageReadRate);
-
- metric(metrics, cluster, namespace, "pulsar_subscription_delayed", stats.msgDelayed);
-
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_msg_backlog", "local", stats.msgBacklog);
-
- stats.managedLedgerStats.storageWriteLatencyBuckets.refresh();
- long[] latencyBuckets = stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets();
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_0_5", latencyBuckets[0]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_1", latencyBuckets[1]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_5", latencyBuckets[2]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_10", latencyBuckets[3]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_20", latencyBuckets[4]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_50", latencyBuckets[5]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_100", latencyBuckets[6]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_200", latencyBuckets[7]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_1000", latencyBuckets[8]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_overflow", latencyBuckets[9]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_count",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getCount());
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_sum",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getSum());
-
- stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.refresh();
- long[] ledgerWritelatencyBuckets = stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets();
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_0_5", ledgerWritelatencyBuckets[0]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_1", ledgerWritelatencyBuckets[1]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_5", ledgerWritelatencyBuckets[2]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_10", ledgerWritelatencyBuckets[3]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_20", ledgerWritelatencyBuckets[4]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_50", ledgerWritelatencyBuckets[5]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_100", ledgerWritelatencyBuckets[6]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_200", ledgerWritelatencyBuckets[7]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_1000",
- ledgerWritelatencyBuckets[8]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_overflow",
- ledgerWritelatencyBuckets[9]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_count",
- stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getCount());
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_sum",
- stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getSum());
-
- stats.managedLedgerStats.entrySizeBuckets.refresh();
- long[] entrySizeBuckets = stats.managedLedgerStats.entrySizeBuckets.getBuckets();
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_128", entrySizeBuckets[0]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_512", entrySizeBuckets[1]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_1_kb", entrySizeBuckets[2]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_2_kb", entrySizeBuckets[3]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_4_kb", entrySizeBuckets[4]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_16_kb", entrySizeBuckets[5]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_100_kb", entrySizeBuckets[6]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_1_mb", entrySizeBuckets[7]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_overflow", entrySizeBuckets[8]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_count",
- stats.managedLedgerStats.entrySizeBuckets.getCount());
- metric(metrics, cluster, namespace, "pulsar_entry_size_sum",
- stats.managedLedgerStats.entrySizeBuckets.getSum());
-
- if (!stats.replicationStats.isEmpty()) {
- stats.replicationStats.forEach((remoteCluster, replStats) -> {
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_rate_in", remoteCluster,
- replStats.msgRateIn);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_rate_out", remoteCluster,
- replStats.msgRateOut);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_throughput_in", remoteCluster,
- replStats.msgThroughputIn);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_throughput_out", remoteCluster,
- replStats.msgThroughputOut);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_backlog", remoteCluster,
- replStats.replicationBacklog);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_connected_count",
- remoteCluster,
- replStats.connectedCount);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_rate_expired", remoteCluster,
- replStats.msgRateExpired);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_delay_in_seconds",
- remoteCluster, replStats.replicationDelayInSeconds);
- });
- }
+ private static void writeMetricWithBrokerDefault(SimpleTextOutputStream stream, String cluster, String name,
+ List<AggregatedNamespaceStats> allNamespaceStats,
+ Function<AggregatedNamespaceStats, Number> namespaceFunction) {
+ stream.write("# TYPE ").write(name).write(" gauge\n");
+ stream.write(name)
+ .write("{cluster=\"").write(cluster).write("\"} ")
+ .write(0).write(' ').write(System.currentTimeMillis())
+ .write('\n');
+ writeNamespaceStats(stream, cluster, name, allNamespaceStats, namespaceFunction);
}
- private static void metric(Map<String, Collector.MetricFamilySamples> metrics, String cluster, String name,
- long value) {
- addMetric(metrics, Map.of("cluster", cluster), name, value);
+ private static void writeMetric(SimpleTextOutputStream stream, String cluster, String name,
+ List<AggregatedNamespaceStats> allNamespaceStats,
+ Function<AggregatedNamespaceStats, Number> namespaceFunction) {
+ stream.write("# TYPE ").write(name).write(" gauge\n");
+ writeNamespaceStats(stream, cluster, name, allNamespaceStats, namespaceFunction);
}
- private static void metric(Map<String, Collector.MetricFamilySamples> metrics, String cluster, String namespace,
- String name, long value) {
- addMetric(metrics, Map.of("cluster", cluster, "namespace", namespace), name, value);
+ private static void writeNamespaceStats(SimpleTextOutputStream stream, String cluster, String name,
+ List<AggregatedNamespaceStats> allNamespaceStats,
+ Function<AggregatedNamespaceStats, Number> namespaceFunction) {
+ allNamespaceStats.forEach(n -> {
+ Number value = namespaceFunction.apply(n);
+ if (value != null) {
+ stream.write(name)
+ .write("{cluster=\"").write(cluster)
+ .write("\",namespace=\"").write(n.name)
+ .write("\"} ")
+ .write(value).write(' ').write(System.currentTimeMillis())
+ .write('\n');
+ }
+ });
}
- private static void metric(Map<String, Collector.MetricFamilySamples> metrics, String cluster, String namespace,
- String name, double value) {
- addMetric(metrics, Map.of("cluster", cluster, "namespace", namespace), name, value);
+ private static void writeMsgBacklog(SimpleTextOutputStream stream, String cluster,
+ List<AggregatedNamespaceStats> allNamespaceStats,
+ Function<AggregatedNamespaceStats, Number> namespaceFunction) {
+ stream.write("# TYPE ").write("pulsar_msg_backlog").write(" gauge\n");
+ stream.write("pulsar_msg_backlog")
+ .write("{cluster=\"").write(cluster).write("\"} ")
+ .write(0).write(' ').write(System.currentTimeMillis())
+ .write('\n');
+ allNamespaceStats.forEach(n -> {
+ Number value = namespaceFunction.apply(n);
+ if (value != null) {
+ stream.write("pulsar_msg_backlog")
+ .write("{cluster=\"").write(cluster)
+ .write("\",namespace=\"").write(n.name)
+ .write("\",remote_cluster=\"").write("local")
+ .write("\"} ")
+ .write(value).write(' ').write(System.currentTimeMillis())
+ .write('\n');
+ }
+ });
}
- private static void metricWithRemoteCluster(Map<String, Collector.MetricFamilySamples> metrics, String cluster,
- String namespace, String name, String remoteCluster, double value) {
- addMetric(metrics, Map.of("cluster", cluster, "namespace", namespace, "remote_cluster", remoteCluster), name,
- value);
+ private static void writeReplicationStat(SimpleTextOutputStream stream, String cluster, String name,
+ List<AggregatedNamespaceStats> allNamespaceStats,
+ Function<AggregatedReplicationStats, Number> replStatsFunction) {
+ stream.write("# TYPE ").write(name).write(" gauge\n");
+ allNamespaceStats.forEach(n -> {
+ if (!n.replicationStats.isEmpty()) {
+ n.replicationStats.forEach((remoteCluster, replStats) ->
+ stream.write(name)
Review Comment:
An idea:
```
writeSample(stream, name, replStatsFunction.apply(replStats), "cluster", cluster, "namespace", n.name, "remote_cluster", remoteCluster)
```
So it's
```
writeSample(stream, String name, Number value, String...labelsAndValuesArray)
```
So we keep the type-safe methods that are good, but try to prevent the same code duplicates?
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/TopicStats.java:
##########
@@ -105,372 +103,457 @@ public void reset() {
compactionLatencyBuckets.reset();
}
- static void resetTypes() {
- metricWithTypeDefinition.clear();
- }
-
- static void buildTopicStats(Map<String, Collector.MetricFamilySamples> metrics, String cluster, String namespace,
- String topic,
- TopicStats stats, Optional<CompactorMXBean> compactorMXBean,
- boolean splitTopicAndPartitionIndexLabel) {
- metric(metrics, cluster, namespace, topic, "pulsar_subscriptions_count", stats.subscriptionsCount,
+ public static void printTopicStats(SimpleTextOutputStream stream, String cluster, List<TopicStats> stats,
+ Optional<CompactorMXBean> compactorMXBean,
+ boolean splitTopicAndPartitionIndexLabel) {
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_subscriptions_count", stats, s -> s.subscriptionsCount,
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_producers_count", stats.producersCount,
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_producers_count", stats, s -> s.producersCount,
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_consumers_count", stats.consumersCount,
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_consumers_count", stats, s -> s.consumersCount,
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_rate_in", stats.rateIn,
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_rate_in", stats, s -> s.rateIn,
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_rate_out", stats.rateOut,
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_rate_out", stats, s -> s.rateOut,
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_throughput_in", stats.throughputIn,
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_throughput_in", stats, s -> s.throughputIn,
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_throughput_out", stats.throughputOut,
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_throughput_out", stats, s -> s.throughputOut,
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_average_msg_size", stats.averageMsgSize,
+ writeMetric(stream, cluster, "pulsar_average_msg_size", stats, s -> s.averageMsgSize,
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_size", stats.managedLedgerStats.storageSize,
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_size", stats,
+ s -> s.managedLedgerStats.storageSize,
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_logical_size",
- stats.managedLedgerStats.storageLogicalSize, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_msg_backlog", stats.msgBacklog,
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_logical_size",
+ stats, s -> s.managedLedgerStats.storageLogicalSize, splitTopicAndPartitionIndexLabel);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_msg_backlog", stats, s -> s.msgBacklog,
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_backlog_size",
- stats.managedLedgerStats.backlogSize, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_publish_rate_limit_times", stats.publishRateLimitedTimes,
+ writeMetric(stream, cluster, "pulsar_storage_backlog_size",
+ stats, s -> s.managedLedgerStats.backlogSize, splitTopicAndPartitionIndexLabel);
+ writeMetric(stream, cluster, "pulsar_publish_rate_limit_times", stats, s -> s.publishRateLimitedTimes,
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_offloaded_size", stats.managedLedgerStats
+ writeMetric(stream, cluster, "pulsar_storage_offloaded_size", stats, s -> s.managedLedgerStats
.offloadedStorageUsed, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_backlog_quota_limit", stats.backlogQuotaLimit,
+ writeMetric(stream, cluster, "pulsar_storage_backlog_quota_limit", stats, s -> s.backlogQuotaLimit,
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_backlog_quota_limit_time",
- stats.backlogQuotaLimitTime, splitTopicAndPartitionIndexLabel);
+ writeMetric(stream, cluster, "pulsar_storage_backlog_quota_limit_time",
+ stats, s -> s.backlogQuotaLimitTime, splitTopicAndPartitionIndexLabel);
- long[] latencyBuckets = stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets();
- metric(metrics, cluster, namespace, topic, "pulsar_storage_write_latency_le_0_5", latencyBuckets[0],
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_0_5", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[0],
+ splitTopicAndPartitionIndexLabel);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_1", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[1],
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_write_latency_le_1", latencyBuckets[1],
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_5", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[2],
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_write_latency_le_5", latencyBuckets[2],
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_10", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[3],
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_write_latency_le_10", latencyBuckets[3],
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_20", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[4],
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_write_latency_le_20", latencyBuckets[4],
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_50", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[5],
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_write_latency_le_50", latencyBuckets[5],
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_100", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[6],
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_write_latency_le_100", latencyBuckets[6],
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_200", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[7],
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_write_latency_le_200", latencyBuckets[7],
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_1000", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[8],
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_write_latency_le_1000", latencyBuckets[8],
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_overflow", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[9],
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_write_latency_overflow", latencyBuckets[9],
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_count",
+ stats, s -> s.managedLedgerStats.storageWriteLatencyBuckets.getCount(),
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_write_latency_count",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getCount(), splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_write_latency_sum",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getSum(), splitTopicAndPartitionIndexLabel);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_sum",
+ stats, s -> s.managedLedgerStats.storageWriteLatencyBuckets.getSum(), splitTopicAndPartitionIndexLabel);
- long[] ledgerWriteLatencyBuckets = stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets();
- metric(metrics, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_0_5",
- ledgerWriteLatencyBuckets[0], splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_1",
- ledgerWriteLatencyBuckets[1], splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_5",
- ledgerWriteLatencyBuckets[2], splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_10",
- ledgerWriteLatencyBuckets[3], splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_20",
- ledgerWriteLatencyBuckets[4], splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_50",
- ledgerWriteLatencyBuckets[5], splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_100",
- ledgerWriteLatencyBuckets[6], splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_200",
- ledgerWriteLatencyBuckets[7], splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_1000",
- ledgerWriteLatencyBuckets[8], splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_overflow",
- ledgerWriteLatencyBuckets[9], splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_count",
- stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getCount(),
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_0_5",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[0],
+ splitTopicAndPartitionIndexLabel);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_1",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[1],
+ splitTopicAndPartitionIndexLabel);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_5",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[2],
+ splitTopicAndPartitionIndexLabel);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_10",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[3],
+ splitTopicAndPartitionIndexLabel);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_20",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[4],
+ splitTopicAndPartitionIndexLabel);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_50",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[5],
+ splitTopicAndPartitionIndexLabel);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_100",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[6],
+ splitTopicAndPartitionIndexLabel);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_200",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[7],
+ splitTopicAndPartitionIndexLabel);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_1000",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[8],
+ splitTopicAndPartitionIndexLabel);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_overflow",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[9],
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_sum",
- stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getSum(),
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_count",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getCount(),
+ splitTopicAndPartitionIndexLabel);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_sum",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getSum(),
splitTopicAndPartitionIndexLabel);
- long[] entrySizeBuckets = stats.managedLedgerStats.entrySizeBuckets.getBuckets();
- metric(metrics, cluster, namespace, topic, "pulsar_entry_size_le_128", entrySizeBuckets[0],
+ writeMetric(stream, cluster, "pulsar_entry_size_le_128", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[0],
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_entry_size_le_512", entrySizeBuckets[1],
+ writeMetric(stream, cluster, "pulsar_entry_size_le_512", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[1],
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_entry_size_le_1_kb", entrySizeBuckets[2],
+ writeMetric(stream, cluster, "pulsar_entry_size_le_1_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[2],
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_entry_size_le_2_kb", entrySizeBuckets[3],
+ writeMetric(stream, cluster, "pulsar_entry_size_le_2_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[3],
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_entry_size_le_4_kb", entrySizeBuckets[4],
+ writeMetric(stream, cluster, "pulsar_entry_size_le_4_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[4],
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_entry_size_le_16_kb", entrySizeBuckets[5],
+ writeMetric(stream, cluster, "pulsar_entry_size_le_16_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[5],
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_entry_size_le_100_kb", entrySizeBuckets[6],
+ writeMetric(stream, cluster, "pulsar_entry_size_le_100_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[6],
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_entry_size_le_1_mb", entrySizeBuckets[7],
+ writeMetric(stream, cluster, "pulsar_entry_size_le_1_mb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[7],
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_entry_size_le_overflow", entrySizeBuckets[8],
+ writeMetric(stream, cluster, "pulsar_entry_size_le_overflow", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[8],
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_entry_size_count",
- stats.managedLedgerStats.entrySizeBuckets.getCount(), splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_entry_size_sum",
- stats.managedLedgerStats.entrySizeBuckets.getSum(), splitTopicAndPartitionIndexLabel);
+ writeMetric(stream, cluster, "pulsar_entry_size_count",
+ stats, s -> s.managedLedgerStats.entrySizeBuckets.getCount(), splitTopicAndPartitionIndexLabel);
+ writeMetric(stream, cluster, "pulsar_entry_size_sum",
+ stats, s -> s.managedLedgerStats.entrySizeBuckets.getSum(), splitTopicAndPartitionIndexLabel);
- stats.producerStats.forEach((p, producerStats) -> {
- metric(metrics, cluster, namespace, topic, p, producerStats.producerId, "pulsar_producer_msg_rate_in",
- producerStats.msgRateIn, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, p, producerStats.producerId, "pulsar_producer_msg_throughput_in",
- producerStats.msgThroughputIn, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, p, producerStats.producerId, "pulsar_producer_msg_average_Size",
- producerStats.averageMsgSize, splitTopicAndPartitionIndexLabel);
- });
+ writeProducerStat(stream, cluster, "pulsar_producer_msg_rate_in", stats,
+ p -> p.msgRateIn, splitTopicAndPartitionIndexLabel);
+ writeProducerStat(stream, cluster, "pulsar_producer_msg_throughput_in", stats,
+ p -> p.msgThroughputIn, splitTopicAndPartitionIndexLabel);
+ writeProducerStat(stream, cluster, "pulsar_producer_msg_average_Size", stats,
+ p -> p.averageMsgSize, splitTopicAndPartitionIndexLabel);
- stats.subscriptionStats.forEach((n, subsStats) -> {
- metric(metrics, cluster, namespace, topic, n, "pulsar_subscription_back_log",
- subsStats.msgBacklog, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, "pulsar_subscription_back_log_no_delayed",
- subsStats.msgBacklogNoDelayed, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, "pulsar_subscription_delayed",
- subsStats.msgDelayed, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, "pulsar_subscription_msg_rate_redeliver",
- subsStats.msgRateRedeliver, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, "pulsar_subscription_unacked_messages",
- subsStats.unackedMessages, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, "pulsar_subscription_blocked_on_unacked_messages",
- subsStats.blockedSubscriptionOnUnackedMsgs ? 1 : 0, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, "pulsar_subscription_msg_rate_out",
- subsStats.msgRateOut, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, "pulsar_subscription_msg_ack_rate",
- subsStats.messageAckRate, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, "pulsar_subscription_msg_throughput_out",
- subsStats.msgThroughputOut, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, "pulsar_out_bytes_total",
- subsStats.bytesOutCounter, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, "pulsar_out_messages_total",
- subsStats.msgOutCounter, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, "pulsar_subscription_last_expire_timestamp",
- subsStats.lastExpireTimestamp, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, "pulsar_subscription_last_acked_timestamp",
- subsStats.lastAckedTimestamp, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, "pulsar_subscription_last_consumed_flow_timestamp",
- subsStats.lastConsumedFlowTimestamp, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, "pulsar_subscription_last_consumed_timestamp",
- subsStats.lastConsumedTimestamp, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, "pulsar_subscription_last_mark_delete_advanced_timestamp",
- subsStats.lastMarkDeleteAdvancedTimestamp, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, "pulsar_subscription_msg_rate_expired",
- subsStats.msgRateExpired, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, "pulsar_subscription_total_msg_expired",
- subsStats.totalMsgExpired, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, "pulsar_subscription_msg_drop_rate",
- subsStats.msgDropRate, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, "pulsar_subscription_consumers_count",
- subsStats.consumersCount, splitTopicAndPartitionIndexLabel);
- subsStats.consumerStat.forEach((c, consumerStats) -> {
- metric(metrics, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_msg_rate_redeliver", consumerStats.msgRateRedeliver,
- splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_unacked_messages", consumerStats.unackedMessages,
- splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_blocked_on_unacked_messages",
- consumerStats.blockedSubscriptionOnUnackedMsgs ? 1 : 0,
- splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_msg_rate_out", consumerStats.msgRateOut,
- splitTopicAndPartitionIndexLabel);
-
- metric(metrics, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_msg_ack_rate", consumerStats.msgAckRate,
- splitTopicAndPartitionIndexLabel);
-
- metric(metrics, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_msg_throughput_out", consumerStats.msgThroughputOut,
- splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_available_permits", consumerStats.availablePermits,
- splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_out_bytes_total", consumerStats.bytesOutCounter,
- splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_out_messages_total", consumerStats.msgOutCounter,
- splitTopicAndPartitionIndexLabel);
- });
- });
- if (!stats.replicationStats.isEmpty()) {
- stats.replicationStats.forEach((remoteCluster, replStats) -> {
- metricWithRemoteCluster(metrics, cluster, namespace, topic, "pulsar_replication_rate_in", remoteCluster,
- replStats.msgRateIn, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(metrics, cluster, namespace, topic, "pulsar_replication_rate_out",
- remoteCluster,
- replStats.msgRateOut, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(metrics, cluster, namespace, topic, "pulsar_replication_throughput_in",
- remoteCluster, replStats.msgThroughputIn, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(metrics, cluster, namespace, topic, "pulsar_replication_throughput_out",
- remoteCluster, replStats.msgThroughputOut, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(metrics, cluster, namespace, topic, "pulsar_replication_backlog", remoteCluster,
- replStats.replicationBacklog, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(metrics, cluster, namespace, topic, "pulsar_replication_connected_count",
- remoteCluster, replStats.connectedCount, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(metrics, cluster, namespace, topic, "pulsar_replication_rate_expired",
- remoteCluster, replStats.msgRateExpired, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(metrics, cluster, namespace, topic, "pulsar_replication_delay_in_seconds",
- remoteCluster, replStats.replicationDelayInSeconds, splitTopicAndPartitionIndexLabel);
- });
- }
+ writeSubscriptionStat(stream, cluster, "pulsar_subscription_back_log", stats,
+ s -> s.msgBacklog, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(stream, cluster, "pulsar_subscription_back_log_no_delayed",
+ stats, s -> s.msgBacklogNoDelayed, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(stream, cluster, "pulsar_subscription_delayed",
+ stats, s -> s.msgDelayed, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(stream, cluster, "pulsar_subscription_msg_rate_redeliver",
+ stats, s -> s.msgRateRedeliver, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(stream, cluster, "pulsar_subscription_unacked_messages",
+ stats, s -> s.unackedMessages, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(stream, cluster, "pulsar_subscription_blocked_on_unacked_messages",
+ stats, s -> s.blockedSubscriptionOnUnackedMsgs ? 1 : 0, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(stream, cluster, "pulsar_subscription_msg_rate_out",
+ stats, s -> s.msgRateOut, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(stream, cluster, "pulsar_subscription_msg_ack_rate",
+ stats, s -> s.messageAckRate, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(stream, cluster, "pulsar_subscription_msg_throughput_out",
+ stats, s -> s.msgThroughputOut, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(stream, cluster, "pulsar_out_bytes_total",
+ stats, s -> s.bytesOutCounter, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(stream, cluster, "pulsar_out_messages_total",
+ stats, s -> s.msgOutCounter, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(stream, cluster, "pulsar_subscription_last_expire_timestamp",
+ stats, s -> s.lastExpireTimestamp, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(stream, cluster, "pulsar_subscription_last_acked_timestamp",
+ stats, s -> s.lastAckedTimestamp, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(stream, cluster, "pulsar_subscription_last_consumed_flow_timestamp",
+ stats, s -> s.lastConsumedFlowTimestamp, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(stream, cluster, "pulsar_subscription_last_consumed_timestamp",
+ stats, s -> s.lastConsumedTimestamp, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(stream, cluster, "pulsar_subscription_last_mark_delete_advanced_timestamp",
+ stats, s -> s.lastMarkDeleteAdvancedTimestamp, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(stream, cluster, "pulsar_subscription_msg_rate_expired",
+ stats, s -> s.msgRateExpired, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(stream, cluster, "pulsar_subscription_total_msg_expired",
+ stats, s -> s.totalMsgExpired, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(stream, cluster, "pulsar_subscription_msg_drop_rate",
+ stats, s -> s.msgDropRate, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(stream, cluster, "pulsar_subscription_consumers_count",
+ stats, s -> s.consumersCount, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_in_bytes_total", stats.bytesInCounter,
+ writeConsumerStat(stream, cluster, "pulsar_consumer_msg_rate_redeliver", stats, c -> c.msgRateRedeliver,
+ splitTopicAndPartitionIndexLabel);
+ writeConsumerStat(stream, cluster, "pulsar_consumer_unacked_messages", stats, c -> c.unackedMessages,
+ splitTopicAndPartitionIndexLabel);
+ writeConsumerStat(stream, cluster, "pulsar_consumer_blocked_on_unacked_messages",
+ stats, c -> c.blockedSubscriptionOnUnackedMsgs ? 1 : 0,
+ splitTopicAndPartitionIndexLabel);
+ writeConsumerStat(stream, cluster, "pulsar_consumer_msg_rate_out", stats, c -> c.msgRateOut,
+ splitTopicAndPartitionIndexLabel);
+
+ writeConsumerStat(stream, cluster, "pulsar_consumer_msg_ack_rate", stats, c -> c.msgAckRate,
+ splitTopicAndPartitionIndexLabel);
+
+ writeConsumerStat(stream, cluster, "pulsar_consumer_msg_throughput_out", stats, c -> c.msgThroughputOut,
+ splitTopicAndPartitionIndexLabel);
+ writeConsumerStat(stream, cluster, "pulsar_consumer_available_permits", stats, c -> c.availablePermits,
+ splitTopicAndPartitionIndexLabel);
+ writeConsumerStat(stream, cluster, "pulsar_out_bytes_total", stats, c -> c.bytesOutCounter,
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_in_messages_total", stats.msgInCounter,
+ writeConsumerStat(stream, cluster, "pulsar_out_messages_total", stats, c -> c.msgOutCounter,
+ splitTopicAndPartitionIndexLabel);
+
+ writeReplicationStat(stream, cluster, "pulsar_replication_rate_in", stats, r -> r.msgRateIn,
+ splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(stream, cluster, "pulsar_replication_rate_out", stats, r -> r.msgRateOut,
+ splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(stream, cluster, "pulsar_replication_throughput_in", stats, r -> r.msgThroughputIn,
+ splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(stream, cluster, "pulsar_replication_throughput_out", stats, r -> r.msgThroughputOut,
+ splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(stream, cluster, "pulsar_replication_backlog", stats, r -> r.replicationBacklog,
+ splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(stream, cluster, "pulsar_replication_connected_count", stats, r -> r.connectedCount,
+ splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(stream, cluster, "pulsar_replication_rate_expired", stats, r -> r.msgRateExpired,
+ splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(stream, cluster, "pulsar_replication_delay_in_seconds", stats,
+ r -> r.replicationDelayInSeconds, splitTopicAndPartitionIndexLabel);
+
+ writeMetric(stream, cluster, "pulsar_in_bytes_total", stats, s -> s.bytesInCounter,
+ splitTopicAndPartitionIndexLabel);
+ writeMetric(stream, cluster, "pulsar_in_messages_total", stats, s -> s.msgInCounter,
splitTopicAndPartitionIndexLabel);
// Compaction
- boolean hasCompaction =
- compactorMXBean.flatMap(mxBean -> mxBean.getCompactionRecordForTopic(topic)).isPresent();
- if (hasCompaction) {
- metric(metrics, cluster, namespace, topic, "pulsar_compaction_removed_event_count",
- stats.compactionRemovedEventCount, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_compaction_succeed_count",
- stats.compactionSucceedCount, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_compaction_failed_count",
- stats.compactionFailedCount, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_compaction_duration_time_in_mills",
- stats.compactionDurationTimeInMills, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_compaction_read_throughput",
- stats.compactionReadThroughput, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_compaction_write_throughput",
- stats.compactionWriteThroughput, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_compaction_compacted_entries_count",
- stats.compactionCompactedEntriesCount, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_compaction_compacted_entries_size",
- stats.compactionCompactedEntriesSize, splitTopicAndPartitionIndexLabel);
- long[] compactionLatencyBuckets = stats.compactionLatencyBuckets.getBuckets();
- metric(metrics, cluster, namespace, topic, "pulsar_compaction_latency_le_0_5",
- compactionLatencyBuckets[0], splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_compaction_latency_le_1",
- compactionLatencyBuckets[1], splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_compaction_latency_le_5",
- compactionLatencyBuckets[2], splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_compaction_latency_le_10",
- compactionLatencyBuckets[3], splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_compaction_latency_le_20",
- compactionLatencyBuckets[4], splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_compaction_latency_le_50",
- compactionLatencyBuckets[5], splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_compaction_latency_le_100",
- compactionLatencyBuckets[6], splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_compaction_latency_le_200",
- compactionLatencyBuckets[7], splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_compaction_latency_le_1000",
- compactionLatencyBuckets[8], splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_compaction_latency_overflow",
- compactionLatencyBuckets[9], splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_compaction_latency_sum",
- stats.compactionLatencyBuckets.getSum(), splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_compaction_latency_count",
- stats.compactionLatencyBuckets.getCount(), splitTopicAndPartitionIndexLabel);
- }
+
+ writeCompactionStat(stream, cluster, "pulsar_compaction_removed_event_count",
+ stats, s -> s.compactionRemovedEventCount, compactorMXBean, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(stream, cluster, "pulsar_compaction_succeed_count",
+ stats, s -> s.compactionSucceedCount, compactorMXBean, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(stream, cluster, "pulsar_compaction_failed_count",
+ stats, s -> s.compactionFailedCount, compactorMXBean, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(stream, cluster, "pulsar_compaction_duration_time_in_mills",
+ stats, s -> s.compactionDurationTimeInMills, compactorMXBean, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(stream, cluster, "pulsar_compaction_read_throughput",
+ stats, s -> s.compactionReadThroughput, compactorMXBean, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(stream, cluster, "pulsar_compaction_write_throughput",
+ stats, s -> s.compactionWriteThroughput, compactorMXBean, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(stream, cluster, "pulsar_compaction_compacted_entries_count",
+ stats, s -> s.compactionCompactedEntriesCount, compactorMXBean, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(stream, cluster, "pulsar_compaction_compacted_entries_size",
+ stats, s -> s.compactionCompactedEntriesSize, compactorMXBean, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(stream, cluster, "pulsar_compaction_latency_le_0_5",
+ stats, s -> s.compactionLatencyBuckets.getBuckets()[0], compactorMXBean,
+ splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(stream, cluster, "pulsar_compaction_latency_le_1",
+ stats, s -> s.compactionLatencyBuckets.getBuckets()[1], compactorMXBean,
+ splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(stream, cluster, "pulsar_compaction_latency_le_5",
+ stats, s -> s.compactionLatencyBuckets.getBuckets()[2], compactorMXBean,
+ splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(stream, cluster, "pulsar_compaction_latency_le_10",
+ stats, s -> s.compactionLatencyBuckets.getBuckets()[3], compactorMXBean,
+ splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(stream, cluster, "pulsar_compaction_latency_le_20",
+ stats, s -> s.compactionLatencyBuckets.getBuckets()[4], compactorMXBean,
+ splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(stream, cluster, "pulsar_compaction_latency_le_50",
+ stats, s -> s.compactionLatencyBuckets.getBuckets()[5], compactorMXBean,
+ splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(stream, cluster, "pulsar_compaction_latency_le_100",
+ stats, s -> s.compactionLatencyBuckets.getBuckets()[6], compactorMXBean,
+ splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(stream, cluster, "pulsar_compaction_latency_le_200",
+ stats, s -> s.compactionLatencyBuckets.getBuckets()[7], compactorMXBean,
+ splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(stream, cluster, "pulsar_compaction_latency_le_1000",
+ stats, s -> s.compactionLatencyBuckets.getBuckets()[8], compactorMXBean,
+ splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(stream, cluster, "pulsar_compaction_latency_overflow",
+ stats, s -> s.compactionLatencyBuckets.getBuckets()[9], compactorMXBean,
+ splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(stream, cluster, "pulsar_compaction_latency_sum",
+ stats, s -> s.compactionLatencyBuckets.getSum(), compactorMXBean, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(stream, cluster, "pulsar_compaction_latency_count",
+ stats, s -> s.compactionLatencyBuckets.getCount(), compactorMXBean, splitTopicAndPartitionIndexLabel);
}
- private static void metric(Map<String, Collector.MetricFamilySamples> metrics, String cluster, String namespace,
- String topic, String name, double value, boolean splitTopicAndPartitionIndexLabel) {
- Map<String, String> labels = buildLabels(cluster, namespace, topic, splitTopicAndPartitionIndexLabel);
- addMetric(metrics, labels, name, value);
+ private static void writeMetricWithBrokerDefault(SimpleTextOutputStream stream, String cluster, String name,
+ List<TopicStats> allTopicStats,
+ Function<TopicStats, Number> topicFunction,
+ boolean splitTopicAndPartitionIndexLabel) {
+ stream.write("# TYPE ").write(name).write(" gauge\n");
+ stream.write(name)
+ .write("{cluster=\"").write(cluster).write("\"} ")
+ .write(0).write(' ').write(System.currentTimeMillis())
+ .write('\n');
+ writeTopicStats(stream, cluster, name, allTopicStats, topicFunction, splitTopicAndPartitionIndexLabel);
}
- private static void metric(Map<String, Collector.MetricFamilySamples> metrics, String cluster, String namespace,
- String topic, String subscription, String name, long value,
- boolean splitTopicAndPartitionIndexLabel) {
- Map<String, String> labels = buildLabels(cluster, namespace, topic, splitTopicAndPartitionIndexLabel);
- labels.put("subscription", subscription);
- addMetric(metrics, labels, name, value);
+ private static void writeMetric(SimpleTextOutputStream stream, String cluster, String name,
+ List<TopicStats> allTopicStats,
+ Function<TopicStats, Number> topicFunction,
+ boolean splitTopicAndPartitionIndexLabel) {
+ stream.write("# TYPE ").write(name).write(" gauge\n");
+ writeTopicStats(stream, cluster, name, allTopicStats, topicFunction, splitTopicAndPartitionIndexLabel);
}
- private static void metric(Map<String, Collector.MetricFamilySamples> metrics, String cluster, String namespace,
- String topic, String producerName, long produceId, String name, double value,
- boolean splitTopicAndPartitionIndexLabel) {
- Map<String, String> labels = buildLabels(cluster, namespace, topic, splitTopicAndPartitionIndexLabel);
- labels.put("producer_name", producerName);
- labels.put("producer_id", String.valueOf(produceId));
- addMetric(metrics, labels, name, value);
+ private static void writeTopicStats(SimpleTextOutputStream stream, String cluster, String name,
+ List<TopicStats> allTopicStats, Function<TopicStats, Number> valueFunction,
+ boolean splitTopicAndPartitionIndexLabel) {
+ allTopicStats.forEach(t -> {
+ writeCommonLabels(stream, cluster, t.namespace, t.name, name, splitTopicAndPartitionIndexLabel);
+ stream.write("\"} ")
+ .write(valueFunction.apply(t)).write(' ')
+ .write(System.currentTimeMillis())
+ .write('\n');
+ });
+ }
+
+ private static void writeProducerStat(SimpleTextOutputStream stream, String cluster, String name,
+ List<TopicStats> allTopicStats,
+ Function<AggregatedProducerStats, Number> valueFunction,
+ boolean splitTopicAndPartitionIndexLabel) {
+ stream.write("# TYPE ").write(name).write(" gauge\n");
+ allTopicStats.forEach(t ->
+ t.producerStats.forEach((p, producerStats) -> {
+ writeCommonLabels(stream, cluster, t.namespace, t.name, name,
+ splitTopicAndPartitionIndexLabel);
+ stream.write("\",producer_name=\"").write(p)
+ .write("\",producer_id=\"").write(String.valueOf(producerStats.producerId))
+ .write("\"} ")
+ .write(valueFunction.apply(producerStats)).write(' ').write(System.currentTimeMillis())
+ .write('\n');
+ }));
}
- private static void metric(Map<String, Collector.MetricFamilySamples> metrics, String cluster, String namespace,
- String topic, String subscription, String name, double value,
- boolean splitTopicAndPartitionIndexLabel) {
- Map<String, String> labels = buildLabels(cluster, namespace, topic, splitTopicAndPartitionIndexLabel);
- labels.put("subscription", subscription);
- addMetric(metrics, labels, name, value);
+ private static void writeSubscriptionStat(SimpleTextOutputStream stream, String cluster, String name,
+ List<TopicStats> allTopicStats,
+ Function<AggregatedSubscriptionStats, Number> valueFunction,
+ boolean splitTopicAndPartitionIndexLabel) {
+ stream.write("# TYPE ").write(name).write(" gauge\n");
+ allTopicStats.forEach(t ->
+ t.subscriptionStats.forEach((s, subStats) -> {
+ writeCommonLabels(stream, cluster, t.namespace, t.name, name,
+ splitTopicAndPartitionIndexLabel);
+ stream.write("\",subscription=\"").write(s)
+ .write("\"} ")
+ .write(valueFunction.apply(subStats)).write(' ').write(System.currentTimeMillis())
+ .write('\n');
+ }));
}
- private static void metric(Map<String, Collector.MetricFamilySamples> metrics, String cluster, String namespace,
- String topic, String subscription, String consumerName, long consumerId, String name,
- double value, boolean splitTopicAndPartitionIndexLabel) {
- Map<String, String> labels = buildLabels(cluster, namespace, topic, splitTopicAndPartitionIndexLabel);
- labels.put("subscription", subscription);
- labels.put("consumer_name", consumerName);
- labels.put("consumer_id", String.valueOf(consumerId));
- addMetric(metrics, labels, name, value);
+ private static void writeConsumerStat(SimpleTextOutputStream stream, String cluster, String name,
+ List<TopicStats> allTopicStats,
+ Function<AggregatedConsumerStats, Number> valueFunction,
+ boolean splitTopicAndPartitionIndexLabel) {
+ stream.write("# TYPE ").write(name).write(" gauge\n");
+ allTopicStats.forEach(t ->
+ t.subscriptionStats.forEach((s, subStats) ->
+ subStats.consumerStat.forEach((c, conStats) -> {
+ writeCommonLabels(stream, cluster, t.namespace, t.name, name,
+ splitTopicAndPartitionIndexLabel);
+ stream.write("\",subscription=\"").write(s)
+ .write("\",consumer_name=\"").write(c.consumerName())
+ .write("\",consumer_id=\"").write(String.valueOf(c.consumerId()))
+ .write("\"} ")
+ .write(valueFunction.apply(conStats)).write(' ').write(System.currentTimeMillis())
+ .write('\n');
+ })));
+ }
+
+
+ private static void writeReplicationStat(SimpleTextOutputStream stream, String cluster, String name,
+ List<TopicStats> allTopicStats,
+ Function<AggregatedReplicationStats, Number> replStatsFunction,
+ boolean splitTopicAndPartitionIndexLabel) {
+ stream.write("# TYPE ").write(name).write(" gauge\n");
+ allTopicStats.forEach(t -> {
+ if (!t.replicationStats.isEmpty()) {
+ t.replicationStats.forEach((remoteCluster, replStats) -> {
+ stream.write(name)
+ .write("{cluster=\"").write(cluster)
+ .write("\",namespace=\"").write(t.namespace);
+ if (splitTopicAndPartitionIndexLabel) {
+ int index = t.name.indexOf(PARTITIONED_TOPIC_SUFFIX);
+ if (index > 0) {
+ stream.write("\",topic=\"").write(t.name.substring(0, index));
+ stream.write("\",partition=\"")
+ .write(t.name.substring(
+ index + PARTITIONED_TOPIC_SUFFIX.length()));
+ } else {
+ stream.write("\",topic=\"").write(t.name);
+ stream.write("\",partition=\"").write("-1");
+ }
+ } else {
+ stream.write("\",topic=\"").write(t.name);
+ }
+ stream.write("\",remote_cluster=\"").write(remoteCluster)
+ .write("\"} ")
+ .write(replStatsFunction.apply(replStats)).write(' ')
+ .write(System.currentTimeMillis())
+ .write('\n');
+ }
+ );
+ }
+ });
}
- private static void metricWithRemoteCluster(Map<String, Collector.MetricFamilySamples> metrics, String cluster,
- String namespace, String topic, String name, String remoteCluster,
- double value, boolean splitTopicAndPartitionIndexLabel) {
- Map<String, String> labels = buildLabels(cluster, namespace, topic, splitTopicAndPartitionIndexLabel);
- labels.put("remote_cluster", remoteCluster);
- addMetric(metrics, labels, name, value);
+ private static void writeCompactionStat(SimpleTextOutputStream stream, String cluster, String name,
+ List<TopicStats> allTopicStats,
+ Function<TopicStats, Number> valueFunction,
+ Optional<CompactorMXBean> compactorMXBean,
+ boolean splitTopicAndPartitionIndexLabel) {
+ stream.write("# TYPE ").write(name).write(" gauge\n");
+ allTopicStats.forEach(t -> {
+ boolean hasCompaction =
+ compactorMXBean.flatMap(mxBean -> mxBean.getCompactionRecordForTopic(t.name))
+ .isPresent();
+ if (hasCompaction) {
+ writeCommonLabels(stream, cluster, t.namespace, t.name, name,
+ splitTopicAndPartitionIndexLabel);
+ stream.write("\"} ")
+ .write(valueFunction.apply(t)).write(' ')
+ .write(System.currentTimeMillis())
+ .write('\n');
+ }
+ }
+ );
}
- private static Map<String, String> buildLabels(String cluster, String namespace, String topic,
- boolean splitTopicAndPartitionIndexLabel) {
- Map<String, String> labels = new HashMap<>();
- labels.put("cluster", cluster);
- labels.put("namespace", namespace);
+ private static void writeCommonLabels(SimpleTextOutputStream stream, String cluster, String namespace, String topic,
+ String metricName, boolean splitTopicAndPartitionIndexLabel) {
+ stream.write(metricName)
Review Comment:
As a continuation to my previous comment on `AggregatedNamespaceStats`, how about we use that `printSample` function, and the actual topic and partition we can store at TopicStats itself
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/TopicStats.java:
##########
@@ -105,372 +103,457 @@ public void reset() {
compactionLatencyBuckets.reset();
}
- static void resetTypes() {
- metricWithTypeDefinition.clear();
- }
-
- static void buildTopicStats(Map<String, Collector.MetricFamilySamples> metrics, String cluster, String namespace,
- String topic,
- TopicStats stats, Optional<CompactorMXBean> compactorMXBean,
- boolean splitTopicAndPartitionIndexLabel) {
- metric(metrics, cluster, namespace, topic, "pulsar_subscriptions_count", stats.subscriptionsCount,
+ public static void printTopicStats(SimpleTextOutputStream stream, String cluster, List<TopicStats> stats,
+ Optional<CompactorMXBean> compactorMXBean,
+ boolean splitTopicAndPartitionIndexLabel) {
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_subscriptions_count", stats, s -> s.subscriptionsCount,
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_producers_count", stats.producersCount,
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_producers_count", stats, s -> s.producersCount,
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_consumers_count", stats.consumersCount,
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_consumers_count", stats, s -> s.consumersCount,
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_rate_in", stats.rateIn,
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_rate_in", stats, s -> s.rateIn,
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_rate_out", stats.rateOut,
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_rate_out", stats, s -> s.rateOut,
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_throughput_in", stats.throughputIn,
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_throughput_in", stats, s -> s.throughputIn,
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_throughput_out", stats.throughputOut,
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_throughput_out", stats, s -> s.throughputOut,
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_average_msg_size", stats.averageMsgSize,
+ writeMetric(stream, cluster, "pulsar_average_msg_size", stats, s -> s.averageMsgSize,
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_size", stats.managedLedgerStats.storageSize,
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_size", stats,
+ s -> s.managedLedgerStats.storageSize,
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_logical_size",
- stats.managedLedgerStats.storageLogicalSize, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_msg_backlog", stats.msgBacklog,
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_logical_size",
+ stats, s -> s.managedLedgerStats.storageLogicalSize, splitTopicAndPartitionIndexLabel);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_msg_backlog", stats, s -> s.msgBacklog,
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_backlog_size",
- stats.managedLedgerStats.backlogSize, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_publish_rate_limit_times", stats.publishRateLimitedTimes,
+ writeMetric(stream, cluster, "pulsar_storage_backlog_size",
+ stats, s -> s.managedLedgerStats.backlogSize, splitTopicAndPartitionIndexLabel);
+ writeMetric(stream, cluster, "pulsar_publish_rate_limit_times", stats, s -> s.publishRateLimitedTimes,
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_offloaded_size", stats.managedLedgerStats
+ writeMetric(stream, cluster, "pulsar_storage_offloaded_size", stats, s -> s.managedLedgerStats
.offloadedStorageUsed, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_backlog_quota_limit", stats.backlogQuotaLimit,
+ writeMetric(stream, cluster, "pulsar_storage_backlog_quota_limit", stats, s -> s.backlogQuotaLimit,
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_backlog_quota_limit_time",
- stats.backlogQuotaLimitTime, splitTopicAndPartitionIndexLabel);
+ writeMetric(stream, cluster, "pulsar_storage_backlog_quota_limit_time",
+ stats, s -> s.backlogQuotaLimitTime, splitTopicAndPartitionIndexLabel);
- long[] latencyBuckets = stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets();
- metric(metrics, cluster, namespace, topic, "pulsar_storage_write_latency_le_0_5", latencyBuckets[0],
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_0_5", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[0],
+ splitTopicAndPartitionIndexLabel);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_1", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[1],
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_write_latency_le_1", latencyBuckets[1],
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_5", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[2],
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_write_latency_le_5", latencyBuckets[2],
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_10", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[3],
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_write_latency_le_10", latencyBuckets[3],
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_20", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[4],
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_write_latency_le_20", latencyBuckets[4],
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_50", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[5],
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_write_latency_le_50", latencyBuckets[5],
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_100", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[6],
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_write_latency_le_100", latencyBuckets[6],
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_200", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[7],
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_write_latency_le_200", latencyBuckets[7],
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_1000", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[8],
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_write_latency_le_1000", latencyBuckets[8],
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_overflow", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[9],
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_write_latency_overflow", latencyBuckets[9],
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_count",
+ stats, s -> s.managedLedgerStats.storageWriteLatencyBuckets.getCount(),
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_write_latency_count",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getCount(), splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_write_latency_sum",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getSum(), splitTopicAndPartitionIndexLabel);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_sum",
+ stats, s -> s.managedLedgerStats.storageWriteLatencyBuckets.getSum(), splitTopicAndPartitionIndexLabel);
- long[] ledgerWriteLatencyBuckets = stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets();
- metric(metrics, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_0_5",
- ledgerWriteLatencyBuckets[0], splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_1",
- ledgerWriteLatencyBuckets[1], splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_5",
- ledgerWriteLatencyBuckets[2], splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_10",
- ledgerWriteLatencyBuckets[3], splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_20",
- ledgerWriteLatencyBuckets[4], splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_50",
- ledgerWriteLatencyBuckets[5], splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_100",
- ledgerWriteLatencyBuckets[6], splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_200",
- ledgerWriteLatencyBuckets[7], splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_1000",
- ledgerWriteLatencyBuckets[8], splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_overflow",
- ledgerWriteLatencyBuckets[9], splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_count",
- stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getCount(),
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_0_5",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[0],
+ splitTopicAndPartitionIndexLabel);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_1",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[1],
+ splitTopicAndPartitionIndexLabel);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_5",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[2],
+ splitTopicAndPartitionIndexLabel);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_10",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[3],
+ splitTopicAndPartitionIndexLabel);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_20",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[4],
+ splitTopicAndPartitionIndexLabel);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_50",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[5],
+ splitTopicAndPartitionIndexLabel);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_100",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[6],
+ splitTopicAndPartitionIndexLabel);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_200",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[7],
+ splitTopicAndPartitionIndexLabel);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_1000",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[8],
+ splitTopicAndPartitionIndexLabel);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_overflow",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[9],
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_sum",
- stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getSum(),
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_count",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getCount(),
+ splitTopicAndPartitionIndexLabel);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_sum",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getSum(),
splitTopicAndPartitionIndexLabel);
- long[] entrySizeBuckets = stats.managedLedgerStats.entrySizeBuckets.getBuckets();
- metric(metrics, cluster, namespace, topic, "pulsar_entry_size_le_128", entrySizeBuckets[0],
+ writeMetric(stream, cluster, "pulsar_entry_size_le_128", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[0],
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_entry_size_le_512", entrySizeBuckets[1],
+ writeMetric(stream, cluster, "pulsar_entry_size_le_512", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[1],
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_entry_size_le_1_kb", entrySizeBuckets[2],
+ writeMetric(stream, cluster, "pulsar_entry_size_le_1_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[2],
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_entry_size_le_2_kb", entrySizeBuckets[3],
+ writeMetric(stream, cluster, "pulsar_entry_size_le_2_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[3],
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_entry_size_le_4_kb", entrySizeBuckets[4],
+ writeMetric(stream, cluster, "pulsar_entry_size_le_4_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[4],
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_entry_size_le_16_kb", entrySizeBuckets[5],
+ writeMetric(stream, cluster, "pulsar_entry_size_le_16_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[5],
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_entry_size_le_100_kb", entrySizeBuckets[6],
+ writeMetric(stream, cluster, "pulsar_entry_size_le_100_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[6],
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_entry_size_le_1_mb", entrySizeBuckets[7],
+ writeMetric(stream, cluster, "pulsar_entry_size_le_1_mb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[7],
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_entry_size_le_overflow", entrySizeBuckets[8],
+ writeMetric(stream, cluster, "pulsar_entry_size_le_overflow", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[8],
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_entry_size_count",
- stats.managedLedgerStats.entrySizeBuckets.getCount(), splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_entry_size_sum",
- stats.managedLedgerStats.entrySizeBuckets.getSum(), splitTopicAndPartitionIndexLabel);
+ writeMetric(stream, cluster, "pulsar_entry_size_count",
+ stats, s -> s.managedLedgerStats.entrySizeBuckets.getCount(), splitTopicAndPartitionIndexLabel);
+ writeMetric(stream, cluster, "pulsar_entry_size_sum",
+ stats, s -> s.managedLedgerStats.entrySizeBuckets.getSum(), splitTopicAndPartitionIndexLabel);
- stats.producerStats.forEach((p, producerStats) -> {
- metric(metrics, cluster, namespace, topic, p, producerStats.producerId, "pulsar_producer_msg_rate_in",
- producerStats.msgRateIn, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, p, producerStats.producerId, "pulsar_producer_msg_throughput_in",
- producerStats.msgThroughputIn, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, p, producerStats.producerId, "pulsar_producer_msg_average_Size",
- producerStats.averageMsgSize, splitTopicAndPartitionIndexLabel);
- });
+ writeProducerStat(stream, cluster, "pulsar_producer_msg_rate_in", stats,
+ p -> p.msgRateIn, splitTopicAndPartitionIndexLabel);
+ writeProducerStat(stream, cluster, "pulsar_producer_msg_throughput_in", stats,
+ p -> p.msgThroughputIn, splitTopicAndPartitionIndexLabel);
+ writeProducerStat(stream, cluster, "pulsar_producer_msg_average_Size", stats,
+ p -> p.averageMsgSize, splitTopicAndPartitionIndexLabel);
- stats.subscriptionStats.forEach((n, subsStats) -> {
- metric(metrics, cluster, namespace, topic, n, "pulsar_subscription_back_log",
- subsStats.msgBacklog, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, "pulsar_subscription_back_log_no_delayed",
- subsStats.msgBacklogNoDelayed, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, "pulsar_subscription_delayed",
- subsStats.msgDelayed, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, "pulsar_subscription_msg_rate_redeliver",
- subsStats.msgRateRedeliver, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, "pulsar_subscription_unacked_messages",
- subsStats.unackedMessages, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, "pulsar_subscription_blocked_on_unacked_messages",
- subsStats.blockedSubscriptionOnUnackedMsgs ? 1 : 0, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, "pulsar_subscription_msg_rate_out",
- subsStats.msgRateOut, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, "pulsar_subscription_msg_ack_rate",
- subsStats.messageAckRate, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, "pulsar_subscription_msg_throughput_out",
- subsStats.msgThroughputOut, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, "pulsar_out_bytes_total",
- subsStats.bytesOutCounter, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, "pulsar_out_messages_total",
- subsStats.msgOutCounter, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, "pulsar_subscription_last_expire_timestamp",
- subsStats.lastExpireTimestamp, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, "pulsar_subscription_last_acked_timestamp",
- subsStats.lastAckedTimestamp, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, "pulsar_subscription_last_consumed_flow_timestamp",
- subsStats.lastConsumedFlowTimestamp, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, "pulsar_subscription_last_consumed_timestamp",
- subsStats.lastConsumedTimestamp, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, "pulsar_subscription_last_mark_delete_advanced_timestamp",
- subsStats.lastMarkDeleteAdvancedTimestamp, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, "pulsar_subscription_msg_rate_expired",
- subsStats.msgRateExpired, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, "pulsar_subscription_total_msg_expired",
- subsStats.totalMsgExpired, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, "pulsar_subscription_msg_drop_rate",
- subsStats.msgDropRate, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, "pulsar_subscription_consumers_count",
- subsStats.consumersCount, splitTopicAndPartitionIndexLabel);
- subsStats.consumerStat.forEach((c, consumerStats) -> {
- metric(metrics, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_msg_rate_redeliver", consumerStats.msgRateRedeliver,
- splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_unacked_messages", consumerStats.unackedMessages,
- splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_blocked_on_unacked_messages",
- consumerStats.blockedSubscriptionOnUnackedMsgs ? 1 : 0,
- splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_msg_rate_out", consumerStats.msgRateOut,
- splitTopicAndPartitionIndexLabel);
-
- metric(metrics, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_msg_ack_rate", consumerStats.msgAckRate,
- splitTopicAndPartitionIndexLabel);
-
- metric(metrics, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_msg_throughput_out", consumerStats.msgThroughputOut,
- splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_available_permits", consumerStats.availablePermits,
- splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_out_bytes_total", consumerStats.bytesOutCounter,
- splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_out_messages_total", consumerStats.msgOutCounter,
- splitTopicAndPartitionIndexLabel);
- });
- });
- if (!stats.replicationStats.isEmpty()) {
- stats.replicationStats.forEach((remoteCluster, replStats) -> {
- metricWithRemoteCluster(metrics, cluster, namespace, topic, "pulsar_replication_rate_in", remoteCluster,
- replStats.msgRateIn, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(metrics, cluster, namespace, topic, "pulsar_replication_rate_out",
- remoteCluster,
- replStats.msgRateOut, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(metrics, cluster, namespace, topic, "pulsar_replication_throughput_in",
- remoteCluster, replStats.msgThroughputIn, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(metrics, cluster, namespace, topic, "pulsar_replication_throughput_out",
- remoteCluster, replStats.msgThroughputOut, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(metrics, cluster, namespace, topic, "pulsar_replication_backlog", remoteCluster,
- replStats.replicationBacklog, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(metrics, cluster, namespace, topic, "pulsar_replication_connected_count",
- remoteCluster, replStats.connectedCount, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(metrics, cluster, namespace, topic, "pulsar_replication_rate_expired",
- remoteCluster, replStats.msgRateExpired, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(metrics, cluster, namespace, topic, "pulsar_replication_delay_in_seconds",
- remoteCluster, replStats.replicationDelayInSeconds, splitTopicAndPartitionIndexLabel);
- });
- }
+ writeSubscriptionStat(stream, cluster, "pulsar_subscription_back_log", stats,
+ s -> s.msgBacklog, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(stream, cluster, "pulsar_subscription_back_log_no_delayed",
+ stats, s -> s.msgBacklogNoDelayed, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(stream, cluster, "pulsar_subscription_delayed",
+ stats, s -> s.msgDelayed, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(stream, cluster, "pulsar_subscription_msg_rate_redeliver",
+ stats, s -> s.msgRateRedeliver, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(stream, cluster, "pulsar_subscription_unacked_messages",
+ stats, s -> s.unackedMessages, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(stream, cluster, "pulsar_subscription_blocked_on_unacked_messages",
+ stats, s -> s.blockedSubscriptionOnUnackedMsgs ? 1 : 0, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(stream, cluster, "pulsar_subscription_msg_rate_out",
+ stats, s -> s.msgRateOut, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(stream, cluster, "pulsar_subscription_msg_ack_rate",
+ stats, s -> s.messageAckRate, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(stream, cluster, "pulsar_subscription_msg_throughput_out",
+ stats, s -> s.msgThroughputOut, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(stream, cluster, "pulsar_out_bytes_total",
+ stats, s -> s.bytesOutCounter, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(stream, cluster, "pulsar_out_messages_total",
+ stats, s -> s.msgOutCounter, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(stream, cluster, "pulsar_subscription_last_expire_timestamp",
+ stats, s -> s.lastExpireTimestamp, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(stream, cluster, "pulsar_subscription_last_acked_timestamp",
+ stats, s -> s.lastAckedTimestamp, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(stream, cluster, "pulsar_subscription_last_consumed_flow_timestamp",
+ stats, s -> s.lastConsumedFlowTimestamp, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(stream, cluster, "pulsar_subscription_last_consumed_timestamp",
+ stats, s -> s.lastConsumedTimestamp, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(stream, cluster, "pulsar_subscription_last_mark_delete_advanced_timestamp",
+ stats, s -> s.lastMarkDeleteAdvancedTimestamp, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(stream, cluster, "pulsar_subscription_msg_rate_expired",
+ stats, s -> s.msgRateExpired, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(stream, cluster, "pulsar_subscription_total_msg_expired",
+ stats, s -> s.totalMsgExpired, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(stream, cluster, "pulsar_subscription_msg_drop_rate",
+ stats, s -> s.msgDropRate, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(stream, cluster, "pulsar_subscription_consumers_count",
+ stats, s -> s.consumersCount, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_in_bytes_total", stats.bytesInCounter,
+ writeConsumerStat(stream, cluster, "pulsar_consumer_msg_rate_redeliver", stats, c -> c.msgRateRedeliver,
+ splitTopicAndPartitionIndexLabel);
+ writeConsumerStat(stream, cluster, "pulsar_consumer_unacked_messages", stats, c -> c.unackedMessages,
+ splitTopicAndPartitionIndexLabel);
+ writeConsumerStat(stream, cluster, "pulsar_consumer_blocked_on_unacked_messages",
+ stats, c -> c.blockedSubscriptionOnUnackedMsgs ? 1 : 0,
+ splitTopicAndPartitionIndexLabel);
+ writeConsumerStat(stream, cluster, "pulsar_consumer_msg_rate_out", stats, c -> c.msgRateOut,
+ splitTopicAndPartitionIndexLabel);
+
+ writeConsumerStat(stream, cluster, "pulsar_consumer_msg_ack_rate", stats, c -> c.msgAckRate,
+ splitTopicAndPartitionIndexLabel);
+
+ writeConsumerStat(stream, cluster, "pulsar_consumer_msg_throughput_out", stats, c -> c.msgThroughputOut,
+ splitTopicAndPartitionIndexLabel);
+ writeConsumerStat(stream, cluster, "pulsar_consumer_available_permits", stats, c -> c.availablePermits,
+ splitTopicAndPartitionIndexLabel);
+ writeConsumerStat(stream, cluster, "pulsar_out_bytes_total", stats, c -> c.bytesOutCounter,
splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_in_messages_total", stats.msgInCounter,
+ writeConsumerStat(stream, cluster, "pulsar_out_messages_total", stats, c -> c.msgOutCounter,
+ splitTopicAndPartitionIndexLabel);
+
+ writeReplicationStat(stream, cluster, "pulsar_replication_rate_in", stats, r -> r.msgRateIn,
+ splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(stream, cluster, "pulsar_replication_rate_out", stats, r -> r.msgRateOut,
+ splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(stream, cluster, "pulsar_replication_throughput_in", stats, r -> r.msgThroughputIn,
+ splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(stream, cluster, "pulsar_replication_throughput_out", stats, r -> r.msgThroughputOut,
+ splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(stream, cluster, "pulsar_replication_backlog", stats, r -> r.replicationBacklog,
+ splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(stream, cluster, "pulsar_replication_connected_count", stats, r -> r.connectedCount,
+ splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(stream, cluster, "pulsar_replication_rate_expired", stats, r -> r.msgRateExpired,
+ splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(stream, cluster, "pulsar_replication_delay_in_seconds", stats,
+ r -> r.replicationDelayInSeconds, splitTopicAndPartitionIndexLabel);
+
+ writeMetric(stream, cluster, "pulsar_in_bytes_total", stats, s -> s.bytesInCounter,
+ splitTopicAndPartitionIndexLabel);
+ writeMetric(stream, cluster, "pulsar_in_messages_total", stats, s -> s.msgInCounter,
splitTopicAndPartitionIndexLabel);
// Compaction
- boolean hasCompaction =
- compactorMXBean.flatMap(mxBean -> mxBean.getCompactionRecordForTopic(topic)).isPresent();
- if (hasCompaction) {
- metric(metrics, cluster, namespace, topic, "pulsar_compaction_removed_event_count",
- stats.compactionRemovedEventCount, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_compaction_succeed_count",
- stats.compactionSucceedCount, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_compaction_failed_count",
- stats.compactionFailedCount, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_compaction_duration_time_in_mills",
- stats.compactionDurationTimeInMills, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_compaction_read_throughput",
- stats.compactionReadThroughput, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_compaction_write_throughput",
- stats.compactionWriteThroughput, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_compaction_compacted_entries_count",
- stats.compactionCompactedEntriesCount, splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_compaction_compacted_entries_size",
- stats.compactionCompactedEntriesSize, splitTopicAndPartitionIndexLabel);
- long[] compactionLatencyBuckets = stats.compactionLatencyBuckets.getBuckets();
- metric(metrics, cluster, namespace, topic, "pulsar_compaction_latency_le_0_5",
- compactionLatencyBuckets[0], splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_compaction_latency_le_1",
- compactionLatencyBuckets[1], splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_compaction_latency_le_5",
- compactionLatencyBuckets[2], splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_compaction_latency_le_10",
- compactionLatencyBuckets[3], splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_compaction_latency_le_20",
- compactionLatencyBuckets[4], splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_compaction_latency_le_50",
- compactionLatencyBuckets[5], splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_compaction_latency_le_100",
- compactionLatencyBuckets[6], splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_compaction_latency_le_200",
- compactionLatencyBuckets[7], splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_compaction_latency_le_1000",
- compactionLatencyBuckets[8], splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_compaction_latency_overflow",
- compactionLatencyBuckets[9], splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_compaction_latency_sum",
- stats.compactionLatencyBuckets.getSum(), splitTopicAndPartitionIndexLabel);
- metric(metrics, cluster, namespace, topic, "pulsar_compaction_latency_count",
- stats.compactionLatencyBuckets.getCount(), splitTopicAndPartitionIndexLabel);
- }
+
+ writeCompactionStat(stream, cluster, "pulsar_compaction_removed_event_count",
+ stats, s -> s.compactionRemovedEventCount, compactorMXBean, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(stream, cluster, "pulsar_compaction_succeed_count",
+ stats, s -> s.compactionSucceedCount, compactorMXBean, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(stream, cluster, "pulsar_compaction_failed_count",
+ stats, s -> s.compactionFailedCount, compactorMXBean, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(stream, cluster, "pulsar_compaction_duration_time_in_mills",
+ stats, s -> s.compactionDurationTimeInMills, compactorMXBean, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(stream, cluster, "pulsar_compaction_read_throughput",
+ stats, s -> s.compactionReadThroughput, compactorMXBean, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(stream, cluster, "pulsar_compaction_write_throughput",
+ stats, s -> s.compactionWriteThroughput, compactorMXBean, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(stream, cluster, "pulsar_compaction_compacted_entries_count",
+ stats, s -> s.compactionCompactedEntriesCount, compactorMXBean, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(stream, cluster, "pulsar_compaction_compacted_entries_size",
+ stats, s -> s.compactionCompactedEntriesSize, compactorMXBean, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(stream, cluster, "pulsar_compaction_latency_le_0_5",
+ stats, s -> s.compactionLatencyBuckets.getBuckets()[0], compactorMXBean,
+ splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(stream, cluster, "pulsar_compaction_latency_le_1",
+ stats, s -> s.compactionLatencyBuckets.getBuckets()[1], compactorMXBean,
+ splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(stream, cluster, "pulsar_compaction_latency_le_5",
+ stats, s -> s.compactionLatencyBuckets.getBuckets()[2], compactorMXBean,
+ splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(stream, cluster, "pulsar_compaction_latency_le_10",
+ stats, s -> s.compactionLatencyBuckets.getBuckets()[3], compactorMXBean,
+ splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(stream, cluster, "pulsar_compaction_latency_le_20",
+ stats, s -> s.compactionLatencyBuckets.getBuckets()[4], compactorMXBean,
+ splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(stream, cluster, "pulsar_compaction_latency_le_50",
+ stats, s -> s.compactionLatencyBuckets.getBuckets()[5], compactorMXBean,
+ splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(stream, cluster, "pulsar_compaction_latency_le_100",
+ stats, s -> s.compactionLatencyBuckets.getBuckets()[6], compactorMXBean,
+ splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(stream, cluster, "pulsar_compaction_latency_le_200",
+ stats, s -> s.compactionLatencyBuckets.getBuckets()[7], compactorMXBean,
+ splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(stream, cluster, "pulsar_compaction_latency_le_1000",
+ stats, s -> s.compactionLatencyBuckets.getBuckets()[8], compactorMXBean,
+ splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(stream, cluster, "pulsar_compaction_latency_overflow",
+ stats, s -> s.compactionLatencyBuckets.getBuckets()[9], compactorMXBean,
+ splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(stream, cluster, "pulsar_compaction_latency_sum",
+ stats, s -> s.compactionLatencyBuckets.getSum(), compactorMXBean, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(stream, cluster, "pulsar_compaction_latency_count",
+ stats, s -> s.compactionLatencyBuckets.getCount(), compactorMXBean, splitTopicAndPartitionIndexLabel);
}
- private static void metric(Map<String, Collector.MetricFamilySamples> metrics, String cluster, String namespace,
- String topic, String name, double value, boolean splitTopicAndPartitionIndexLabel) {
- Map<String, String> labels = buildLabels(cluster, namespace, topic, splitTopicAndPartitionIndexLabel);
- addMetric(metrics, labels, name, value);
+ private static void writeMetricWithBrokerDefault(SimpleTextOutputStream stream, String cluster, String name,
+ List<TopicStats> allTopicStats,
+ Function<TopicStats, Number> topicFunction,
+ boolean splitTopicAndPartitionIndexLabel) {
+ stream.write("# TYPE ").write(name).write(" gauge\n");
+ stream.write(name)
+ .write("{cluster=\"").write(cluster).write("\"} ")
+ .write(0).write(' ').write(System.currentTimeMillis())
+ .write('\n');
+ writeTopicStats(stream, cluster, name, allTopicStats, topicFunction, splitTopicAndPartitionIndexLabel);
}
- private static void metric(Map<String, Collector.MetricFamilySamples> metrics, String cluster, String namespace,
- String topic, String subscription, String name, long value,
- boolean splitTopicAndPartitionIndexLabel) {
- Map<String, String> labels = buildLabels(cluster, namespace, topic, splitTopicAndPartitionIndexLabel);
- labels.put("subscription", subscription);
- addMetric(metrics, labels, name, value);
+ private static void writeMetric(SimpleTextOutputStream stream, String cluster, String name,
+ List<TopicStats> allTopicStats,
+ Function<TopicStats, Number> topicFunction,
+ boolean splitTopicAndPartitionIndexLabel) {
+ stream.write("# TYPE ").write(name).write(" gauge\n");
+ writeTopicStats(stream, cluster, name, allTopicStats, topicFunction, splitTopicAndPartitionIndexLabel);
}
- private static void metric(Map<String, Collector.MetricFamilySamples> metrics, String cluster, String namespace,
- String topic, String producerName, long produceId, String name, double value,
- boolean splitTopicAndPartitionIndexLabel) {
- Map<String, String> labels = buildLabels(cluster, namespace, topic, splitTopicAndPartitionIndexLabel);
- labels.put("producer_name", producerName);
- labels.put("producer_id", String.valueOf(produceId));
- addMetric(metrics, labels, name, value);
+ private static void writeTopicStats(SimpleTextOutputStream stream, String cluster, String name,
+ List<TopicStats> allTopicStats, Function<TopicStats, Number> valueFunction,
+ boolean splitTopicAndPartitionIndexLabel) {
+ allTopicStats.forEach(t -> {
+ writeCommonLabels(stream, cluster, t.namespace, t.name, name, splitTopicAndPartitionIndexLabel);
+ stream.write("\"} ")
+ .write(valueFunction.apply(t)).write(' ')
+ .write(System.currentTimeMillis())
+ .write('\n');
+ });
+ }
+
+ private static void writeProducerStat(SimpleTextOutputStream stream, String cluster, String name,
+ List<TopicStats> allTopicStats,
+ Function<AggregatedProducerStats, Number> valueFunction,
+ boolean splitTopicAndPartitionIndexLabel) {
+ stream.write("# TYPE ").write(name).write(" gauge\n");
+ allTopicStats.forEach(t ->
+ t.producerStats.forEach((p, producerStats) -> {
+ writeCommonLabels(stream, cluster, t.namespace, t.name, name,
+ splitTopicAndPartitionIndexLabel);
+ stream.write("\",producer_name=\"").write(p)
+ .write("\",producer_id=\"").write(String.valueOf(producerStats.producerId))
Review Comment:
no need to `valueOf` this `long` as `SimpleTextOutputStream` has one for `long`, thus we can prevent this malloc and free.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/NamespaceStatsAggregator.java:
##########
@@ -305,155 +332,224 @@ private static void getTopicStats(Topic topic, TopicStats stats, boolean include
});
}
- private static void buildDefaultBrokerStats(Map<String, Collector.MetricFamilySamples> metrics, String cluster) {
- // Print metrics with 0 values. This is necessary to have the available brokers being
- // reported in the brokers dashboard even if they don't have any topic or traffic
- metric(metrics, cluster, "pulsar_topics_count", 0);
- metric(metrics, cluster, "pulsar_subscriptions_count", 0);
- metric(metrics, cluster, "pulsar_producers_count", 0);
- metric(metrics, cluster, "pulsar_consumers_count", 0);
- metric(metrics, cluster, "pulsar_rate_in", 0);
- metric(metrics, cluster, "pulsar_rate_out", 0);
- metric(metrics, cluster, "pulsar_throughput_in", 0);
- metric(metrics, cluster, "pulsar_throughput_out", 0);
- metric(metrics, cluster, "pulsar_storage_size", 0);
- metric(metrics, cluster, "pulsar_storage_logical_size", 0);
- metric(metrics, cluster, "pulsar_storage_write_rate", 0);
- metric(metrics, cluster, "pulsar_storage_read_rate", 0);
- metric(metrics, cluster, "pulsar_msg_backlog", 0);
+ private static void printTopicsCountStats(SimpleTextOutputStream stream, String cluster,
+ Map<String, Long> namespaceTopicsCount) {
+ stream.write("# TYPE ").write("pulsar_topics_count").write(" gauge\n");
+ stream.write("pulsar_topics_count")
+ .write("{cluster=\"").write(cluster).write("\"} ")
+ .write(0).write(' ').write(System.currentTimeMillis())
+ .write('\n');
+ namespaceTopicsCount.forEach((ns, topicCount) -> stream.write("pulsar_topics_count")
+ .write("{cluster=\"").write(cluster)
+ .write("\",namespace=\"").write(ns)
+ .write("\"} ")
+ .write(topicCount).write(' ').write(System.currentTimeMillis())
+ .write('\n')
+ );
}
- private static void printTopicsCountStats(Map<String, Collector.MetricFamilySamples> metrics, String cluster,
- String namespace,
- LongAdder topicsCount) {
- metric(metrics, cluster, namespace, "pulsar_topics_count", topicsCount.sum());
+ private static void printNamespaceStats(SimpleTextOutputStream stream, String cluster,
+ List<AggregatedNamespaceStats> stats) {
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_topics_count", stats, s -> s.topicsCount);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_subscriptions_count", stats, s -> s.subscriptionsCount);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_producers_count", stats, s -> s.producersCount);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_consumers_count", stats, s -> s.consumersCount);
+
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_rate_in", stats, s -> s.rateIn);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_rate_out", stats, s -> s.rateOut);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_throughput_in", stats, s -> s.throughputIn);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_throughput_out", stats, s -> s.throughputOut);
+ writeMetric(stream, cluster, "pulsar_consumer_msg_ack_rate", stats, s -> s.messageAckRate);
+
+ writeMetric(stream, cluster, "pulsar_in_bytes_total", stats, s -> s.bytesInCounter);
+ writeMetric(stream, cluster, "pulsar_in_messages_total", stats, s -> s.msgInCounter);
+ writeMetric(stream, cluster, "pulsar_out_bytes_total", stats, s -> s.bytesOutCounter);
+ writeMetric(stream, cluster, "pulsar_out_messages_total", stats, s -> s.msgOutCounter);
+
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_size", stats,
+ s -> s.managedLedgerStats.storageSize);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_logical_size", stats,
+ s -> s.managedLedgerStats.storageLogicalSize);
+ writeMetric(stream, cluster, "pulsar_storage_backlog_size", stats, s -> s.managedLedgerStats.backlogSize);
+ writeMetric(stream, cluster, "pulsar_storage_offloaded_size",
+ stats, s -> s.managedLedgerStats.offloadedStorageUsed);
+
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_write_rate", stats,
+ s -> s.managedLedgerStats.storageWriteRate);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_read_rate", stats,
+ s -> s.managedLedgerStats.storageReadRate);
+
+ writeMetric(stream, cluster, "pulsar_subscription_delayed", stats, s -> s.msgDelayed);
+
+ writeMsgBacklog(stream, cluster, stats, s -> s.msgBacklog);
+
+ stats.forEach(s -> s.managedLedgerStats.storageWriteLatencyBuckets.refresh());
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_0_5", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[0]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_1", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[1]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_5", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[2]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_10", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[3]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_20", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[4]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_50", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[5]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_100", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[6]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_200", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[7]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_1000", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[8]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_overflow", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[9]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_count",
+ stats, s -> s.managedLedgerStats.storageWriteLatencyBuckets.getCount());
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_sum",
+ stats, s -> s.managedLedgerStats.storageWriteLatencyBuckets.getSum());
+
+ stats.forEach(s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.refresh());
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_0_5", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[0]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_1", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[1]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_5", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[2]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_10", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[3]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_20", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[4]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_50", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[5]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_100", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[6]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_200", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[7]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_1000",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[8]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_overflow",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[9]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_count",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getCount());
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_sum",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getSum());
+
+ stats.forEach(s -> s.managedLedgerStats.entrySizeBuckets.refresh());
+ writeMetric(stream, cluster, "pulsar_entry_size_le_128", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[0]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_512", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[1]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_1_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[2]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_2_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[3]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_4_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[4]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_16_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[5]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_100_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[6]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_1_mb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[7]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_overflow", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[8]);
+ writeMetric(stream, cluster, "pulsar_entry_size_count",
+ stats, s -> s.managedLedgerStats.entrySizeBuckets.getCount());
+ writeMetric(stream, cluster, "pulsar_entry_size_sum",
+ stats, s -> s.managedLedgerStats.entrySizeBuckets.getSum());
+
+ writeReplicationStat(stream, cluster, "pulsar_replication_rate_in", stats,
+ replStats -> replStats.msgRateIn);
+ writeReplicationStat(stream, cluster, "pulsar_replication_rate_out", stats,
+ replStats -> replStats.msgRateOut);
+ writeReplicationStat(stream, cluster, "pulsar_replication_throughput_in", stats,
+ replStats -> replStats.msgThroughputIn);
+ writeReplicationStat(stream, cluster, "pulsar_replication_throughput_out", stats,
+ replStats -> replStats.msgThroughputOut);
+ writeReplicationStat(stream, cluster, "pulsar_replication_backlog", stats,
+ replStats -> replStats.replicationBacklog);
+ writeReplicationStat(stream, cluster, "pulsar_replication_connected_count", stats,
+ replStats -> replStats.connectedCount);
+ writeReplicationStat(stream, cluster, "pulsar_replication_rate_expired", stats,
+ replStats -> replStats.msgRateExpired);
+ writeReplicationStat(stream, cluster, "pulsar_replication_delay_in_seconds", stats,
+ replStats -> replStats.replicationDelayInSeconds);
}
- private static void printNamespaceStats(Map<String, Collector.MetricFamilySamples> metrics, String cluster,
- String namespace,
- AggregatedNamespaceStats stats) {
- metric(metrics, cluster, namespace, "pulsar_topics_count", stats.topicsCount);
- metric(metrics, cluster, namespace, "pulsar_subscriptions_count", stats.subscriptionsCount);
- metric(metrics, cluster, namespace, "pulsar_producers_count", stats.producersCount);
- metric(metrics, cluster, namespace, "pulsar_consumers_count", stats.consumersCount);
-
- metric(metrics, cluster, namespace, "pulsar_rate_in", stats.rateIn);
- metric(metrics, cluster, namespace, "pulsar_rate_out", stats.rateOut);
- metric(metrics, cluster, namespace, "pulsar_throughput_in", stats.throughputIn);
- metric(metrics, cluster, namespace, "pulsar_throughput_out", stats.throughputOut);
- metric(metrics, cluster, namespace, "pulsar_consumer_msg_ack_rate", stats.messageAckRate);
-
- metric(metrics, cluster, namespace, "pulsar_in_bytes_total", stats.bytesInCounter);
- metric(metrics, cluster, namespace, "pulsar_in_messages_total", stats.msgInCounter);
- metric(metrics, cluster, namespace, "pulsar_out_bytes_total", stats.bytesOutCounter);
- metric(metrics, cluster, namespace, "pulsar_out_messages_total", stats.msgOutCounter);
-
- metric(metrics, cluster, namespace, "pulsar_storage_size", stats.managedLedgerStats.storageSize);
- metric(metrics, cluster, namespace, "pulsar_storage_logical_size", stats.managedLedgerStats.storageLogicalSize);
- metric(metrics, cluster, namespace, "pulsar_storage_backlog_size", stats.managedLedgerStats.backlogSize);
- metric(metrics, cluster, namespace, "pulsar_storage_offloaded_size",
- stats.managedLedgerStats.offloadedStorageUsed);
-
- metric(metrics, cluster, namespace, "pulsar_storage_write_rate", stats.managedLedgerStats.storageWriteRate);
- metric(metrics, cluster, namespace, "pulsar_storage_read_rate", stats.managedLedgerStats.storageReadRate);
-
- metric(metrics, cluster, namespace, "pulsar_subscription_delayed", stats.msgDelayed);
-
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_msg_backlog", "local", stats.msgBacklog);
-
- stats.managedLedgerStats.storageWriteLatencyBuckets.refresh();
- long[] latencyBuckets = stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets();
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_0_5", latencyBuckets[0]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_1", latencyBuckets[1]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_5", latencyBuckets[2]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_10", latencyBuckets[3]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_20", latencyBuckets[4]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_50", latencyBuckets[5]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_100", latencyBuckets[6]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_200", latencyBuckets[7]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_1000", latencyBuckets[8]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_overflow", latencyBuckets[9]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_count",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getCount());
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_sum",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getSum());
-
- stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.refresh();
- long[] ledgerWritelatencyBuckets = stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets();
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_0_5", ledgerWritelatencyBuckets[0]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_1", ledgerWritelatencyBuckets[1]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_5", ledgerWritelatencyBuckets[2]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_10", ledgerWritelatencyBuckets[3]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_20", ledgerWritelatencyBuckets[4]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_50", ledgerWritelatencyBuckets[5]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_100", ledgerWritelatencyBuckets[6]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_200", ledgerWritelatencyBuckets[7]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_1000",
- ledgerWritelatencyBuckets[8]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_overflow",
- ledgerWritelatencyBuckets[9]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_count",
- stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getCount());
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_sum",
- stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getSum());
-
- stats.managedLedgerStats.entrySizeBuckets.refresh();
- long[] entrySizeBuckets = stats.managedLedgerStats.entrySizeBuckets.getBuckets();
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_128", entrySizeBuckets[0]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_512", entrySizeBuckets[1]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_1_kb", entrySizeBuckets[2]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_2_kb", entrySizeBuckets[3]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_4_kb", entrySizeBuckets[4]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_16_kb", entrySizeBuckets[5]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_100_kb", entrySizeBuckets[6]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_1_mb", entrySizeBuckets[7]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_overflow", entrySizeBuckets[8]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_count",
- stats.managedLedgerStats.entrySizeBuckets.getCount());
- metric(metrics, cluster, namespace, "pulsar_entry_size_sum",
- stats.managedLedgerStats.entrySizeBuckets.getSum());
-
- if (!stats.replicationStats.isEmpty()) {
- stats.replicationStats.forEach((remoteCluster, replStats) -> {
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_rate_in", remoteCluster,
- replStats.msgRateIn);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_rate_out", remoteCluster,
- replStats.msgRateOut);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_throughput_in", remoteCluster,
- replStats.msgThroughputIn);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_throughput_out", remoteCluster,
- replStats.msgThroughputOut);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_backlog", remoteCluster,
- replStats.replicationBacklog);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_connected_count",
- remoteCluster,
- replStats.connectedCount);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_rate_expired", remoteCluster,
- replStats.msgRateExpired);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_delay_in_seconds",
- remoteCluster, replStats.replicationDelayInSeconds);
- });
- }
+ private static void writeMetricWithBrokerDefault(SimpleTextOutputStream stream, String cluster, String name,
+ List<AggregatedNamespaceStats> allNamespaceStats,
+ Function<AggregatedNamespaceStats, Number> namespaceFunction) {
+ stream.write("# TYPE ").write(name).write(" gauge\n");
+ stream.write(name)
+ .write("{cluster=\"").write(cluster).write("\"} ")
+ .write(0).write(' ').write(System.currentTimeMillis())
+ .write('\n');
+ writeNamespaceStats(stream, cluster, name, allNamespaceStats, namespaceFunction);
Review Comment:
Maybe `writeNamespaceStats` --> `writeAllNamespacesStats` ?
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/NamespaceStatsAggregator.java:
##########
@@ -305,155 +332,224 @@ private static void getTopicStats(Topic topic, TopicStats stats, boolean include
});
}
- private static void buildDefaultBrokerStats(Map<String, Collector.MetricFamilySamples> metrics, String cluster) {
- // Print metrics with 0 values. This is necessary to have the available brokers being
- // reported in the brokers dashboard even if they don't have any topic or traffic
- metric(metrics, cluster, "pulsar_topics_count", 0);
- metric(metrics, cluster, "pulsar_subscriptions_count", 0);
- metric(metrics, cluster, "pulsar_producers_count", 0);
- metric(metrics, cluster, "pulsar_consumers_count", 0);
- metric(metrics, cluster, "pulsar_rate_in", 0);
- metric(metrics, cluster, "pulsar_rate_out", 0);
- metric(metrics, cluster, "pulsar_throughput_in", 0);
- metric(metrics, cluster, "pulsar_throughput_out", 0);
- metric(metrics, cluster, "pulsar_storage_size", 0);
- metric(metrics, cluster, "pulsar_storage_logical_size", 0);
- metric(metrics, cluster, "pulsar_storage_write_rate", 0);
- metric(metrics, cluster, "pulsar_storage_read_rate", 0);
- metric(metrics, cluster, "pulsar_msg_backlog", 0);
+ private static void printTopicsCountStats(SimpleTextOutputStream stream, String cluster,
+ Map<String, Long> namespaceTopicsCount) {
+ stream.write("# TYPE ").write("pulsar_topics_count").write(" gauge\n");
+ stream.write("pulsar_topics_count")
+ .write("{cluster=\"").write(cluster).write("\"} ")
+ .write(0).write(' ').write(System.currentTimeMillis())
+ .write('\n');
+ namespaceTopicsCount.forEach((ns, topicCount) -> stream.write("pulsar_topics_count")
+ .write("{cluster=\"").write(cluster)
+ .write("\",namespace=\"").write(ns)
+ .write("\"} ")
+ .write(topicCount).write(' ').write(System.currentTimeMillis())
+ .write('\n')
+ );
}
- private static void printTopicsCountStats(Map<String, Collector.MetricFamilySamples> metrics, String cluster,
- String namespace,
- LongAdder topicsCount) {
- metric(metrics, cluster, namespace, "pulsar_topics_count", topicsCount.sum());
+ private static void printNamespaceStats(SimpleTextOutputStream stream, String cluster,
+ List<AggregatedNamespaceStats> stats) {
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_topics_count", stats, s -> s.topicsCount);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_subscriptions_count", stats, s -> s.subscriptionsCount);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_producers_count", stats, s -> s.producersCount);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_consumers_count", stats, s -> s.consumersCount);
+
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_rate_in", stats, s -> s.rateIn);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_rate_out", stats, s -> s.rateOut);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_throughput_in", stats, s -> s.throughputIn);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_throughput_out", stats, s -> s.throughputOut);
+ writeMetric(stream, cluster, "pulsar_consumer_msg_ack_rate", stats, s -> s.messageAckRate);
+
+ writeMetric(stream, cluster, "pulsar_in_bytes_total", stats, s -> s.bytesInCounter);
+ writeMetric(stream, cluster, "pulsar_in_messages_total", stats, s -> s.msgInCounter);
+ writeMetric(stream, cluster, "pulsar_out_bytes_total", stats, s -> s.bytesOutCounter);
+ writeMetric(stream, cluster, "pulsar_out_messages_total", stats, s -> s.msgOutCounter);
+
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_size", stats,
+ s -> s.managedLedgerStats.storageSize);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_logical_size", stats,
+ s -> s.managedLedgerStats.storageLogicalSize);
+ writeMetric(stream, cluster, "pulsar_storage_backlog_size", stats, s -> s.managedLedgerStats.backlogSize);
+ writeMetric(stream, cluster, "pulsar_storage_offloaded_size",
+ stats, s -> s.managedLedgerStats.offloadedStorageUsed);
+
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_write_rate", stats,
+ s -> s.managedLedgerStats.storageWriteRate);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_read_rate", stats,
+ s -> s.managedLedgerStats.storageReadRate);
+
+ writeMetric(stream, cluster, "pulsar_subscription_delayed", stats, s -> s.msgDelayed);
+
+ writeMsgBacklog(stream, cluster, stats, s -> s.msgBacklog);
+
+ stats.forEach(s -> s.managedLedgerStats.storageWriteLatencyBuckets.refresh());
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_0_5", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[0]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_1", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[1]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_5", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[2]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_10", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[3]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_20", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[4]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_50", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[5]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_100", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[6]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_200", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[7]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_1000", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[8]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_overflow", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[9]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_count",
+ stats, s -> s.managedLedgerStats.storageWriteLatencyBuckets.getCount());
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_sum",
+ stats, s -> s.managedLedgerStats.storageWriteLatencyBuckets.getSum());
+
+ stats.forEach(s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.refresh());
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_0_5", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[0]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_1", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[1]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_5", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[2]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_10", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[3]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_20", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[4]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_50", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[5]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_100", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[6]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_200", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[7]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_1000",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[8]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_overflow",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[9]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_count",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getCount());
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_sum",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getSum());
+
+ stats.forEach(s -> s.managedLedgerStats.entrySizeBuckets.refresh());
+ writeMetric(stream, cluster, "pulsar_entry_size_le_128", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[0]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_512", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[1]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_1_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[2]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_2_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[3]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_4_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[4]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_16_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[5]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_100_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[6]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_1_mb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[7]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_overflow", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[8]);
+ writeMetric(stream, cluster, "pulsar_entry_size_count",
+ stats, s -> s.managedLedgerStats.entrySizeBuckets.getCount());
+ writeMetric(stream, cluster, "pulsar_entry_size_sum",
+ stats, s -> s.managedLedgerStats.entrySizeBuckets.getSum());
+
+ writeReplicationStat(stream, cluster, "pulsar_replication_rate_in", stats,
+ replStats -> replStats.msgRateIn);
+ writeReplicationStat(stream, cluster, "pulsar_replication_rate_out", stats,
+ replStats -> replStats.msgRateOut);
+ writeReplicationStat(stream, cluster, "pulsar_replication_throughput_in", stats,
+ replStats -> replStats.msgThroughputIn);
+ writeReplicationStat(stream, cluster, "pulsar_replication_throughput_out", stats,
+ replStats -> replStats.msgThroughputOut);
+ writeReplicationStat(stream, cluster, "pulsar_replication_backlog", stats,
+ replStats -> replStats.replicationBacklog);
+ writeReplicationStat(stream, cluster, "pulsar_replication_connected_count", stats,
+ replStats -> replStats.connectedCount);
+ writeReplicationStat(stream, cluster, "pulsar_replication_rate_expired", stats,
+ replStats -> replStats.msgRateExpired);
+ writeReplicationStat(stream, cluster, "pulsar_replication_delay_in_seconds", stats,
+ replStats -> replStats.replicationDelayInSeconds);
}
- private static void printNamespaceStats(Map<String, Collector.MetricFamilySamples> metrics, String cluster,
- String namespace,
- AggregatedNamespaceStats stats) {
- metric(metrics, cluster, namespace, "pulsar_topics_count", stats.topicsCount);
- metric(metrics, cluster, namespace, "pulsar_subscriptions_count", stats.subscriptionsCount);
- metric(metrics, cluster, namespace, "pulsar_producers_count", stats.producersCount);
- metric(metrics, cluster, namespace, "pulsar_consumers_count", stats.consumersCount);
-
- metric(metrics, cluster, namespace, "pulsar_rate_in", stats.rateIn);
- metric(metrics, cluster, namespace, "pulsar_rate_out", stats.rateOut);
- metric(metrics, cluster, namespace, "pulsar_throughput_in", stats.throughputIn);
- metric(metrics, cluster, namespace, "pulsar_throughput_out", stats.throughputOut);
- metric(metrics, cluster, namespace, "pulsar_consumer_msg_ack_rate", stats.messageAckRate);
-
- metric(metrics, cluster, namespace, "pulsar_in_bytes_total", stats.bytesInCounter);
- metric(metrics, cluster, namespace, "pulsar_in_messages_total", stats.msgInCounter);
- metric(metrics, cluster, namespace, "pulsar_out_bytes_total", stats.bytesOutCounter);
- metric(metrics, cluster, namespace, "pulsar_out_messages_total", stats.msgOutCounter);
-
- metric(metrics, cluster, namespace, "pulsar_storage_size", stats.managedLedgerStats.storageSize);
- metric(metrics, cluster, namespace, "pulsar_storage_logical_size", stats.managedLedgerStats.storageLogicalSize);
- metric(metrics, cluster, namespace, "pulsar_storage_backlog_size", stats.managedLedgerStats.backlogSize);
- metric(metrics, cluster, namespace, "pulsar_storage_offloaded_size",
- stats.managedLedgerStats.offloadedStorageUsed);
-
- metric(metrics, cluster, namespace, "pulsar_storage_write_rate", stats.managedLedgerStats.storageWriteRate);
- metric(metrics, cluster, namespace, "pulsar_storage_read_rate", stats.managedLedgerStats.storageReadRate);
-
- metric(metrics, cluster, namespace, "pulsar_subscription_delayed", stats.msgDelayed);
-
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_msg_backlog", "local", stats.msgBacklog);
-
- stats.managedLedgerStats.storageWriteLatencyBuckets.refresh();
- long[] latencyBuckets = stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets();
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_0_5", latencyBuckets[0]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_1", latencyBuckets[1]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_5", latencyBuckets[2]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_10", latencyBuckets[3]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_20", latencyBuckets[4]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_50", latencyBuckets[5]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_100", latencyBuckets[6]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_200", latencyBuckets[7]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_1000", latencyBuckets[8]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_overflow", latencyBuckets[9]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_count",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getCount());
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_sum",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getSum());
-
- stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.refresh();
- long[] ledgerWritelatencyBuckets = stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets();
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_0_5", ledgerWritelatencyBuckets[0]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_1", ledgerWritelatencyBuckets[1]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_5", ledgerWritelatencyBuckets[2]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_10", ledgerWritelatencyBuckets[3]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_20", ledgerWritelatencyBuckets[4]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_50", ledgerWritelatencyBuckets[5]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_100", ledgerWritelatencyBuckets[6]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_200", ledgerWritelatencyBuckets[7]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_1000",
- ledgerWritelatencyBuckets[8]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_overflow",
- ledgerWritelatencyBuckets[9]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_count",
- stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getCount());
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_sum",
- stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getSum());
-
- stats.managedLedgerStats.entrySizeBuckets.refresh();
- long[] entrySizeBuckets = stats.managedLedgerStats.entrySizeBuckets.getBuckets();
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_128", entrySizeBuckets[0]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_512", entrySizeBuckets[1]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_1_kb", entrySizeBuckets[2]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_2_kb", entrySizeBuckets[3]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_4_kb", entrySizeBuckets[4]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_16_kb", entrySizeBuckets[5]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_100_kb", entrySizeBuckets[6]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_1_mb", entrySizeBuckets[7]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_overflow", entrySizeBuckets[8]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_count",
- stats.managedLedgerStats.entrySizeBuckets.getCount());
- metric(metrics, cluster, namespace, "pulsar_entry_size_sum",
- stats.managedLedgerStats.entrySizeBuckets.getSum());
-
- if (!stats.replicationStats.isEmpty()) {
- stats.replicationStats.forEach((remoteCluster, replStats) -> {
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_rate_in", remoteCluster,
- replStats.msgRateIn);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_rate_out", remoteCluster,
- replStats.msgRateOut);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_throughput_in", remoteCluster,
- replStats.msgThroughputIn);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_throughput_out", remoteCluster,
- replStats.msgThroughputOut);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_backlog", remoteCluster,
- replStats.replicationBacklog);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_connected_count",
- remoteCluster,
- replStats.connectedCount);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_rate_expired", remoteCluster,
- replStats.msgRateExpired);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_delay_in_seconds",
- remoteCluster, replStats.replicationDelayInSeconds);
- });
- }
+ private static void writeMetricWithBrokerDefault(SimpleTextOutputStream stream, String cluster, String name,
Review Comment:
Great improvement
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/NamespaceStatsAggregator.java:
##########
@@ -305,155 +332,224 @@ private static void getTopicStats(Topic topic, TopicStats stats, boolean include
});
}
- private static void buildDefaultBrokerStats(Map<String, Collector.MetricFamilySamples> metrics, String cluster) {
- // Print metrics with 0 values. This is necessary to have the available brokers being
- // reported in the brokers dashboard even if they don't have any topic or traffic
- metric(metrics, cluster, "pulsar_topics_count", 0);
- metric(metrics, cluster, "pulsar_subscriptions_count", 0);
- metric(metrics, cluster, "pulsar_producers_count", 0);
- metric(metrics, cluster, "pulsar_consumers_count", 0);
- metric(metrics, cluster, "pulsar_rate_in", 0);
- metric(metrics, cluster, "pulsar_rate_out", 0);
- metric(metrics, cluster, "pulsar_throughput_in", 0);
- metric(metrics, cluster, "pulsar_throughput_out", 0);
- metric(metrics, cluster, "pulsar_storage_size", 0);
- metric(metrics, cluster, "pulsar_storage_logical_size", 0);
- metric(metrics, cluster, "pulsar_storage_write_rate", 0);
- metric(metrics, cluster, "pulsar_storage_read_rate", 0);
- metric(metrics, cluster, "pulsar_msg_backlog", 0);
+ private static void printTopicsCountStats(SimpleTextOutputStream stream, String cluster,
+ Map<String, Long> namespaceTopicsCount) {
+ stream.write("# TYPE ").write("pulsar_topics_count").write(" gauge\n");
+ stream.write("pulsar_topics_count")
+ .write("{cluster=\"").write(cluster).write("\"} ")
+ .write(0).write(' ').write(System.currentTimeMillis())
+ .write('\n');
+ namespaceTopicsCount.forEach((ns, topicCount) -> stream.write("pulsar_topics_count")
+ .write("{cluster=\"").write(cluster)
+ .write("\",namespace=\"").write(ns)
+ .write("\"} ")
+ .write(topicCount).write(' ').write(System.currentTimeMillis())
+ .write('\n')
+ );
}
- private static void printTopicsCountStats(Map<String, Collector.MetricFamilySamples> metrics, String cluster,
- String namespace,
- LongAdder topicsCount) {
- metric(metrics, cluster, namespace, "pulsar_topics_count", topicsCount.sum());
+ private static void printNamespaceStats(SimpleTextOutputStream stream, String cluster,
+ List<AggregatedNamespaceStats> stats) {
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_topics_count", stats, s -> s.topicsCount);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_subscriptions_count", stats, s -> s.subscriptionsCount);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_producers_count", stats, s -> s.producersCount);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_consumers_count", stats, s -> s.consumersCount);
+
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_rate_in", stats, s -> s.rateIn);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_rate_out", stats, s -> s.rateOut);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_throughput_in", stats, s -> s.throughputIn);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_throughput_out", stats, s -> s.throughputOut);
+ writeMetric(stream, cluster, "pulsar_consumer_msg_ack_rate", stats, s -> s.messageAckRate);
+
+ writeMetric(stream, cluster, "pulsar_in_bytes_total", stats, s -> s.bytesInCounter);
+ writeMetric(stream, cluster, "pulsar_in_messages_total", stats, s -> s.msgInCounter);
+ writeMetric(stream, cluster, "pulsar_out_bytes_total", stats, s -> s.bytesOutCounter);
+ writeMetric(stream, cluster, "pulsar_out_messages_total", stats, s -> s.msgOutCounter);
+
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_size", stats,
+ s -> s.managedLedgerStats.storageSize);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_logical_size", stats,
+ s -> s.managedLedgerStats.storageLogicalSize);
+ writeMetric(stream, cluster, "pulsar_storage_backlog_size", stats, s -> s.managedLedgerStats.backlogSize);
+ writeMetric(stream, cluster, "pulsar_storage_offloaded_size",
+ stats, s -> s.managedLedgerStats.offloadedStorageUsed);
+
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_write_rate", stats,
+ s -> s.managedLedgerStats.storageWriteRate);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_read_rate", stats,
+ s -> s.managedLedgerStats.storageReadRate);
+
+ writeMetric(stream, cluster, "pulsar_subscription_delayed", stats, s -> s.msgDelayed);
+
+ writeMsgBacklog(stream, cluster, stats, s -> s.msgBacklog);
+
+ stats.forEach(s -> s.managedLedgerStats.storageWriteLatencyBuckets.refresh());
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_0_5", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[0]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_1", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[1]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_5", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[2]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_10", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[3]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_20", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[4]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_50", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[5]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_100", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[6]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_200", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[7]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_1000", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[8]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_overflow", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[9]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_count",
+ stats, s -> s.managedLedgerStats.storageWriteLatencyBuckets.getCount());
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_sum",
+ stats, s -> s.managedLedgerStats.storageWriteLatencyBuckets.getSum());
+
+ stats.forEach(s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.refresh());
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_0_5", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[0]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_1", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[1]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_5", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[2]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_10", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[3]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_20", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[4]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_50", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[5]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_100", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[6]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_200", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[7]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_1000",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[8]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_overflow",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[9]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_count",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getCount());
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_sum",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getSum());
+
+ stats.forEach(s -> s.managedLedgerStats.entrySizeBuckets.refresh());
+ writeMetric(stream, cluster, "pulsar_entry_size_le_128", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[0]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_512", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[1]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_1_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[2]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_2_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[3]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_4_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[4]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_16_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[5]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_100_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[6]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_1_mb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[7]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_overflow", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[8]);
+ writeMetric(stream, cluster, "pulsar_entry_size_count",
+ stats, s -> s.managedLedgerStats.entrySizeBuckets.getCount());
+ writeMetric(stream, cluster, "pulsar_entry_size_sum",
+ stats, s -> s.managedLedgerStats.entrySizeBuckets.getSum());
+
+ writeReplicationStat(stream, cluster, "pulsar_replication_rate_in", stats,
+ replStats -> replStats.msgRateIn);
+ writeReplicationStat(stream, cluster, "pulsar_replication_rate_out", stats,
+ replStats -> replStats.msgRateOut);
+ writeReplicationStat(stream, cluster, "pulsar_replication_throughput_in", stats,
+ replStats -> replStats.msgThroughputIn);
+ writeReplicationStat(stream, cluster, "pulsar_replication_throughput_out", stats,
+ replStats -> replStats.msgThroughputOut);
+ writeReplicationStat(stream, cluster, "pulsar_replication_backlog", stats,
+ replStats -> replStats.replicationBacklog);
+ writeReplicationStat(stream, cluster, "pulsar_replication_connected_count", stats,
+ replStats -> replStats.connectedCount);
+ writeReplicationStat(stream, cluster, "pulsar_replication_rate_expired", stats,
+ replStats -> replStats.msgRateExpired);
+ writeReplicationStat(stream, cluster, "pulsar_replication_delay_in_seconds", stats,
+ replStats -> replStats.replicationDelayInSeconds);
}
- private static void printNamespaceStats(Map<String, Collector.MetricFamilySamples> metrics, String cluster,
- String namespace,
- AggregatedNamespaceStats stats) {
- metric(metrics, cluster, namespace, "pulsar_topics_count", stats.topicsCount);
- metric(metrics, cluster, namespace, "pulsar_subscriptions_count", stats.subscriptionsCount);
- metric(metrics, cluster, namespace, "pulsar_producers_count", stats.producersCount);
- metric(metrics, cluster, namespace, "pulsar_consumers_count", stats.consumersCount);
-
- metric(metrics, cluster, namespace, "pulsar_rate_in", stats.rateIn);
- metric(metrics, cluster, namespace, "pulsar_rate_out", stats.rateOut);
- metric(metrics, cluster, namespace, "pulsar_throughput_in", stats.throughputIn);
- metric(metrics, cluster, namespace, "pulsar_throughput_out", stats.throughputOut);
- metric(metrics, cluster, namespace, "pulsar_consumer_msg_ack_rate", stats.messageAckRate);
-
- metric(metrics, cluster, namespace, "pulsar_in_bytes_total", stats.bytesInCounter);
- metric(metrics, cluster, namespace, "pulsar_in_messages_total", stats.msgInCounter);
- metric(metrics, cluster, namespace, "pulsar_out_bytes_total", stats.bytesOutCounter);
- metric(metrics, cluster, namespace, "pulsar_out_messages_total", stats.msgOutCounter);
-
- metric(metrics, cluster, namespace, "pulsar_storage_size", stats.managedLedgerStats.storageSize);
- metric(metrics, cluster, namespace, "pulsar_storage_logical_size", stats.managedLedgerStats.storageLogicalSize);
- metric(metrics, cluster, namespace, "pulsar_storage_backlog_size", stats.managedLedgerStats.backlogSize);
- metric(metrics, cluster, namespace, "pulsar_storage_offloaded_size",
- stats.managedLedgerStats.offloadedStorageUsed);
-
- metric(metrics, cluster, namespace, "pulsar_storage_write_rate", stats.managedLedgerStats.storageWriteRate);
- metric(metrics, cluster, namespace, "pulsar_storage_read_rate", stats.managedLedgerStats.storageReadRate);
-
- metric(metrics, cluster, namespace, "pulsar_subscription_delayed", stats.msgDelayed);
-
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_msg_backlog", "local", stats.msgBacklog);
-
- stats.managedLedgerStats.storageWriteLatencyBuckets.refresh();
- long[] latencyBuckets = stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets();
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_0_5", latencyBuckets[0]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_1", latencyBuckets[1]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_5", latencyBuckets[2]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_10", latencyBuckets[3]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_20", latencyBuckets[4]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_50", latencyBuckets[5]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_100", latencyBuckets[6]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_200", latencyBuckets[7]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_1000", latencyBuckets[8]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_overflow", latencyBuckets[9]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_count",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getCount());
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_sum",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getSum());
-
- stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.refresh();
- long[] ledgerWritelatencyBuckets = stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets();
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_0_5", ledgerWritelatencyBuckets[0]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_1", ledgerWritelatencyBuckets[1]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_5", ledgerWritelatencyBuckets[2]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_10", ledgerWritelatencyBuckets[3]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_20", ledgerWritelatencyBuckets[4]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_50", ledgerWritelatencyBuckets[5]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_100", ledgerWritelatencyBuckets[6]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_200", ledgerWritelatencyBuckets[7]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_1000",
- ledgerWritelatencyBuckets[8]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_overflow",
- ledgerWritelatencyBuckets[9]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_count",
- stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getCount());
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_sum",
- stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getSum());
-
- stats.managedLedgerStats.entrySizeBuckets.refresh();
- long[] entrySizeBuckets = stats.managedLedgerStats.entrySizeBuckets.getBuckets();
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_128", entrySizeBuckets[0]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_512", entrySizeBuckets[1]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_1_kb", entrySizeBuckets[2]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_2_kb", entrySizeBuckets[3]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_4_kb", entrySizeBuckets[4]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_16_kb", entrySizeBuckets[5]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_100_kb", entrySizeBuckets[6]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_1_mb", entrySizeBuckets[7]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_overflow", entrySizeBuckets[8]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_count",
- stats.managedLedgerStats.entrySizeBuckets.getCount());
- metric(metrics, cluster, namespace, "pulsar_entry_size_sum",
- stats.managedLedgerStats.entrySizeBuckets.getSum());
-
- if (!stats.replicationStats.isEmpty()) {
- stats.replicationStats.forEach((remoteCluster, replStats) -> {
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_rate_in", remoteCluster,
- replStats.msgRateIn);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_rate_out", remoteCluster,
- replStats.msgRateOut);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_throughput_in", remoteCluster,
- replStats.msgThroughputIn);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_throughput_out", remoteCluster,
- replStats.msgThroughputOut);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_backlog", remoteCluster,
- replStats.replicationBacklog);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_connected_count",
- remoteCluster,
- replStats.connectedCount);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_rate_expired", remoteCluster,
- replStats.msgRateExpired);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_delay_in_seconds",
- remoteCluster, replStats.replicationDelayInSeconds);
- });
- }
+ private static void writeMetricWithBrokerDefault(SimpleTextOutputStream stream, String cluster, String name,
+ List<AggregatedNamespaceStats> allNamespaceStats,
+ Function<AggregatedNamespaceStats, Number> namespaceFunction) {
+ stream.write("# TYPE ").write(name).write(" gauge\n");
+ stream.write(name)
+ .write("{cluster=\"").write(cluster).write("\"} ")
+ .write(0).write(' ').write(System.currentTimeMillis())
+ .write('\n');
+ writeNamespaceStats(stream, cluster, name, allNamespaceStats, namespaceFunction);
}
- private static void metric(Map<String, Collector.MetricFamilySamples> metrics, String cluster, String name,
- long value) {
- addMetric(metrics, Map.of("cluster", cluster), name, value);
+ private static void writeMetric(SimpleTextOutputStream stream, String cluster, String name,
+ List<AggregatedNamespaceStats> allNamespaceStats,
+ Function<AggregatedNamespaceStats, Number> namespaceFunction) {
+ stream.write("# TYPE ").write(name).write(" gauge\n");
+ writeNamespaceStats(stream, cluster, name, allNamespaceStats, namespaceFunction);
}
- private static void metric(Map<String, Collector.MetricFamilySamples> metrics, String cluster, String namespace,
- String name, long value) {
- addMetric(metrics, Map.of("cluster", cluster, "namespace", namespace), name, value);
+ private static void writeNamespaceStats(SimpleTextOutputStream stream, String cluster, String name,
+ List<AggregatedNamespaceStats> allNamespaceStats,
+ Function<AggregatedNamespaceStats, Number> namespaceFunction) {
+ allNamespaceStats.forEach(n -> {
+ Number value = namespaceFunction.apply(n);
+ if (value != null) {
+ stream.write(name)
+ .write("{cluster=\"").write(cluster)
+ .write("\",namespace=\"").write(n.name)
+ .write("\"} ")
+ .write(value).write(' ').write(System.currentTimeMillis())
+ .write('\n');
+ }
+ });
}
- private static void metric(Map<String, Collector.MetricFamilySamples> metrics, String cluster, String namespace,
- String name, double value) {
- addMetric(metrics, Map.of("cluster", cluster, "namespace", namespace), name, value);
+ private static void writeMsgBacklog(SimpleTextOutputStream stream, String cluster,
+ List<AggregatedNamespaceStats> allNamespaceStats,
+ Function<AggregatedNamespaceStats, Number> namespaceFunction) {
+ stream.write("# TYPE ").write("pulsar_msg_backlog").write(" gauge\n");
+ stream.write("pulsar_msg_backlog")
Review Comment:
Only if there are no namespaces
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/NamespaceStatsAggregator.java:
##########
@@ -305,155 +332,224 @@ private static void getTopicStats(Topic topic, TopicStats stats, boolean include
});
}
- private static void buildDefaultBrokerStats(Map<String, Collector.MetricFamilySamples> metrics, String cluster) {
- // Print metrics with 0 values. This is necessary to have the available brokers being
- // reported in the brokers dashboard even if they don't have any topic or traffic
- metric(metrics, cluster, "pulsar_topics_count", 0);
- metric(metrics, cluster, "pulsar_subscriptions_count", 0);
- metric(metrics, cluster, "pulsar_producers_count", 0);
- metric(metrics, cluster, "pulsar_consumers_count", 0);
- metric(metrics, cluster, "pulsar_rate_in", 0);
- metric(metrics, cluster, "pulsar_rate_out", 0);
- metric(metrics, cluster, "pulsar_throughput_in", 0);
- metric(metrics, cluster, "pulsar_throughput_out", 0);
- metric(metrics, cluster, "pulsar_storage_size", 0);
- metric(metrics, cluster, "pulsar_storage_logical_size", 0);
- metric(metrics, cluster, "pulsar_storage_write_rate", 0);
- metric(metrics, cluster, "pulsar_storage_read_rate", 0);
- metric(metrics, cluster, "pulsar_msg_backlog", 0);
+ private static void printTopicsCountStats(SimpleTextOutputStream stream, String cluster,
+ Map<String, Long> namespaceTopicsCount) {
+ stream.write("# TYPE ").write("pulsar_topics_count").write(" gauge\n");
+ stream.write("pulsar_topics_count")
+ .write("{cluster=\"").write(cluster).write("\"} ")
+ .write(0).write(' ').write(System.currentTimeMillis())
+ .write('\n');
+ namespaceTopicsCount.forEach((ns, topicCount) -> stream.write("pulsar_topics_count")
+ .write("{cluster=\"").write(cluster)
+ .write("\",namespace=\"").write(ns)
+ .write("\"} ")
+ .write(topicCount).write(' ').write(System.currentTimeMillis())
+ .write('\n')
+ );
}
- private static void printTopicsCountStats(Map<String, Collector.MetricFamilySamples> metrics, String cluster,
- String namespace,
- LongAdder topicsCount) {
- metric(metrics, cluster, namespace, "pulsar_topics_count", topicsCount.sum());
+ private static void printNamespaceStats(SimpleTextOutputStream stream, String cluster,
+ List<AggregatedNamespaceStats> stats) {
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_topics_count", stats, s -> s.topicsCount);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_subscriptions_count", stats, s -> s.subscriptionsCount);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_producers_count", stats, s -> s.producersCount);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_consumers_count", stats, s -> s.consumersCount);
+
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_rate_in", stats, s -> s.rateIn);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_rate_out", stats, s -> s.rateOut);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_throughput_in", stats, s -> s.throughputIn);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_throughput_out", stats, s -> s.throughputOut);
+ writeMetric(stream, cluster, "pulsar_consumer_msg_ack_rate", stats, s -> s.messageAckRate);
+
+ writeMetric(stream, cluster, "pulsar_in_bytes_total", stats, s -> s.bytesInCounter);
+ writeMetric(stream, cluster, "pulsar_in_messages_total", stats, s -> s.msgInCounter);
+ writeMetric(stream, cluster, "pulsar_out_bytes_total", stats, s -> s.bytesOutCounter);
+ writeMetric(stream, cluster, "pulsar_out_messages_total", stats, s -> s.msgOutCounter);
+
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_size", stats,
+ s -> s.managedLedgerStats.storageSize);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_logical_size", stats,
+ s -> s.managedLedgerStats.storageLogicalSize);
+ writeMetric(stream, cluster, "pulsar_storage_backlog_size", stats, s -> s.managedLedgerStats.backlogSize);
+ writeMetric(stream, cluster, "pulsar_storage_offloaded_size",
+ stats, s -> s.managedLedgerStats.offloadedStorageUsed);
+
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_write_rate", stats,
+ s -> s.managedLedgerStats.storageWriteRate);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_read_rate", stats,
+ s -> s.managedLedgerStats.storageReadRate);
+
+ writeMetric(stream, cluster, "pulsar_subscription_delayed", stats, s -> s.msgDelayed);
+
+ writeMsgBacklog(stream, cluster, stats, s -> s.msgBacklog);
+
+ stats.forEach(s -> s.managedLedgerStats.storageWriteLatencyBuckets.refresh());
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_0_5", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[0]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_1", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[1]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_5", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[2]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_10", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[3]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_20", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[4]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_50", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[5]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_100", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[6]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_200", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[7]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_1000", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[8]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_overflow", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[9]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_count",
+ stats, s -> s.managedLedgerStats.storageWriteLatencyBuckets.getCount());
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_sum",
+ stats, s -> s.managedLedgerStats.storageWriteLatencyBuckets.getSum());
+
+ stats.forEach(s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.refresh());
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_0_5", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[0]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_1", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[1]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_5", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[2]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_10", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[3]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_20", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[4]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_50", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[5]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_100", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[6]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_200", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[7]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_1000",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[8]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_overflow",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[9]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_count",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getCount());
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_sum",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getSum());
+
+ stats.forEach(s -> s.managedLedgerStats.entrySizeBuckets.refresh());
+ writeMetric(stream, cluster, "pulsar_entry_size_le_128", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[0]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_512", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[1]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_1_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[2]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_2_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[3]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_4_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[4]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_16_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[5]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_100_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[6]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_1_mb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[7]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_overflow", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[8]);
+ writeMetric(stream, cluster, "pulsar_entry_size_count",
+ stats, s -> s.managedLedgerStats.entrySizeBuckets.getCount());
+ writeMetric(stream, cluster, "pulsar_entry_size_sum",
+ stats, s -> s.managedLedgerStats.entrySizeBuckets.getSum());
+
+ writeReplicationStat(stream, cluster, "pulsar_replication_rate_in", stats,
+ replStats -> replStats.msgRateIn);
+ writeReplicationStat(stream, cluster, "pulsar_replication_rate_out", stats,
+ replStats -> replStats.msgRateOut);
+ writeReplicationStat(stream, cluster, "pulsar_replication_throughput_in", stats,
+ replStats -> replStats.msgThroughputIn);
+ writeReplicationStat(stream, cluster, "pulsar_replication_throughput_out", stats,
+ replStats -> replStats.msgThroughputOut);
+ writeReplicationStat(stream, cluster, "pulsar_replication_backlog", stats,
+ replStats -> replStats.replicationBacklog);
+ writeReplicationStat(stream, cluster, "pulsar_replication_connected_count", stats,
+ replStats -> replStats.connectedCount);
+ writeReplicationStat(stream, cluster, "pulsar_replication_rate_expired", stats,
+ replStats -> replStats.msgRateExpired);
+ writeReplicationStat(stream, cluster, "pulsar_replication_delay_in_seconds", stats,
+ replStats -> replStats.replicationDelayInSeconds);
}
- private static void printNamespaceStats(Map<String, Collector.MetricFamilySamples> metrics, String cluster,
- String namespace,
- AggregatedNamespaceStats stats) {
- metric(metrics, cluster, namespace, "pulsar_topics_count", stats.topicsCount);
- metric(metrics, cluster, namespace, "pulsar_subscriptions_count", stats.subscriptionsCount);
- metric(metrics, cluster, namespace, "pulsar_producers_count", stats.producersCount);
- metric(metrics, cluster, namespace, "pulsar_consumers_count", stats.consumersCount);
-
- metric(metrics, cluster, namespace, "pulsar_rate_in", stats.rateIn);
- metric(metrics, cluster, namespace, "pulsar_rate_out", stats.rateOut);
- metric(metrics, cluster, namespace, "pulsar_throughput_in", stats.throughputIn);
- metric(metrics, cluster, namespace, "pulsar_throughput_out", stats.throughputOut);
- metric(metrics, cluster, namespace, "pulsar_consumer_msg_ack_rate", stats.messageAckRate);
-
- metric(metrics, cluster, namespace, "pulsar_in_bytes_total", stats.bytesInCounter);
- metric(metrics, cluster, namespace, "pulsar_in_messages_total", stats.msgInCounter);
- metric(metrics, cluster, namespace, "pulsar_out_bytes_total", stats.bytesOutCounter);
- metric(metrics, cluster, namespace, "pulsar_out_messages_total", stats.msgOutCounter);
-
- metric(metrics, cluster, namespace, "pulsar_storage_size", stats.managedLedgerStats.storageSize);
- metric(metrics, cluster, namespace, "pulsar_storage_logical_size", stats.managedLedgerStats.storageLogicalSize);
- metric(metrics, cluster, namespace, "pulsar_storage_backlog_size", stats.managedLedgerStats.backlogSize);
- metric(metrics, cluster, namespace, "pulsar_storage_offloaded_size",
- stats.managedLedgerStats.offloadedStorageUsed);
-
- metric(metrics, cluster, namespace, "pulsar_storage_write_rate", stats.managedLedgerStats.storageWriteRate);
- metric(metrics, cluster, namespace, "pulsar_storage_read_rate", stats.managedLedgerStats.storageReadRate);
-
- metric(metrics, cluster, namespace, "pulsar_subscription_delayed", stats.msgDelayed);
-
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_msg_backlog", "local", stats.msgBacklog);
-
- stats.managedLedgerStats.storageWriteLatencyBuckets.refresh();
- long[] latencyBuckets = stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets();
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_0_5", latencyBuckets[0]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_1", latencyBuckets[1]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_5", latencyBuckets[2]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_10", latencyBuckets[3]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_20", latencyBuckets[4]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_50", latencyBuckets[5]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_100", latencyBuckets[6]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_200", latencyBuckets[7]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_1000", latencyBuckets[8]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_overflow", latencyBuckets[9]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_count",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getCount());
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_sum",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getSum());
-
- stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.refresh();
- long[] ledgerWritelatencyBuckets = stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets();
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_0_5", ledgerWritelatencyBuckets[0]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_1", ledgerWritelatencyBuckets[1]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_5", ledgerWritelatencyBuckets[2]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_10", ledgerWritelatencyBuckets[3]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_20", ledgerWritelatencyBuckets[4]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_50", ledgerWritelatencyBuckets[5]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_100", ledgerWritelatencyBuckets[6]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_200", ledgerWritelatencyBuckets[7]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_1000",
- ledgerWritelatencyBuckets[8]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_overflow",
- ledgerWritelatencyBuckets[9]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_count",
- stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getCount());
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_sum",
- stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getSum());
-
- stats.managedLedgerStats.entrySizeBuckets.refresh();
- long[] entrySizeBuckets = stats.managedLedgerStats.entrySizeBuckets.getBuckets();
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_128", entrySizeBuckets[0]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_512", entrySizeBuckets[1]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_1_kb", entrySizeBuckets[2]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_2_kb", entrySizeBuckets[3]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_4_kb", entrySizeBuckets[4]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_16_kb", entrySizeBuckets[5]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_100_kb", entrySizeBuckets[6]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_1_mb", entrySizeBuckets[7]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_overflow", entrySizeBuckets[8]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_count",
- stats.managedLedgerStats.entrySizeBuckets.getCount());
- metric(metrics, cluster, namespace, "pulsar_entry_size_sum",
- stats.managedLedgerStats.entrySizeBuckets.getSum());
-
- if (!stats.replicationStats.isEmpty()) {
- stats.replicationStats.forEach((remoteCluster, replStats) -> {
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_rate_in", remoteCluster,
- replStats.msgRateIn);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_rate_out", remoteCluster,
- replStats.msgRateOut);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_throughput_in", remoteCluster,
- replStats.msgThroughputIn);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_throughput_out", remoteCluster,
- replStats.msgThroughputOut);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_backlog", remoteCluster,
- replStats.replicationBacklog);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_connected_count",
- remoteCluster,
- replStats.connectedCount);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_rate_expired", remoteCluster,
- replStats.msgRateExpired);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_delay_in_seconds",
- remoteCluster, replStats.replicationDelayInSeconds);
- });
- }
+ private static void writeMetricWithBrokerDefault(SimpleTextOutputStream stream, String cluster, String name,
+ List<AggregatedNamespaceStats> allNamespaceStats,
+ Function<AggregatedNamespaceStats, Number> namespaceFunction) {
+ stream.write("# TYPE ").write(name).write(" gauge\n");
+ stream.write(name)
+ .write("{cluster=\"").write(cluster).write("\"} ")
+ .write(0).write(' ').write(System.currentTimeMillis())
+ .write('\n');
+ writeNamespaceStats(stream, cluster, name, allNamespaceStats, namespaceFunction);
}
- private static void metric(Map<String, Collector.MetricFamilySamples> metrics, String cluster, String name,
- long value) {
- addMetric(metrics, Map.of("cluster", cluster), name, value);
+ private static void writeMetric(SimpleTextOutputStream stream, String cluster, String name,
+ List<AggregatedNamespaceStats> allNamespaceStats,
+ Function<AggregatedNamespaceStats, Number> namespaceFunction) {
+ stream.write("# TYPE ").write(name).write(" gauge\n");
+ writeNamespaceStats(stream, cluster, name, allNamespaceStats, namespaceFunction);
}
- private static void metric(Map<String, Collector.MetricFamilySamples> metrics, String cluster, String namespace,
- String name, long value) {
- addMetric(metrics, Map.of("cluster", cluster, "namespace", namespace), name, value);
+ private static void writeNamespaceStats(SimpleTextOutputStream stream, String cluster, String name,
+ List<AggregatedNamespaceStats> allNamespaceStats,
+ Function<AggregatedNamespaceStats, Number> namespaceFunction) {
+ allNamespaceStats.forEach(n -> {
+ Number value = namespaceFunction.apply(n);
+ if (value != null) {
+ stream.write(name)
+ .write("{cluster=\"").write(cluster)
+ .write("\",namespace=\"").write(n.name)
+ .write("\"} ")
+ .write(value).write(' ').write(System.currentTimeMillis())
+ .write('\n');
+ }
+ });
}
- private static void metric(Map<String, Collector.MetricFamilySamples> metrics, String cluster, String namespace,
- String name, double value) {
- addMetric(metrics, Map.of("cluster", cluster, "namespace", namespace), name, value);
+ private static void writeMsgBacklog(SimpleTextOutputStream stream, String cluster,
+ List<AggregatedNamespaceStats> allNamespaceStats,
+ Function<AggregatedNamespaceStats, Number> namespaceFunction) {
+ stream.write("# TYPE ").write("pulsar_msg_backlog").write(" gauge\n");
+ stream.write("pulsar_msg_backlog")
+ .write("{cluster=\"").write(cluster).write("\"} ")
+ .write(0).write(' ').write(System.currentTimeMillis())
+ .write('\n');
+ allNamespaceStats.forEach(n -> {
+ Number value = namespaceFunction.apply(n);
+ if (value != null) {
+ stream.write("pulsar_msg_backlog")
+ .write("{cluster=\"").write(cluster)
+ .write("\",namespace=\"").write(n.name)
+ .write("\",remote_cluster=\"").write("local")
+ .write("\"} ")
+ .write(value).write(' ').write(System.currentTimeMillis())
+ .write('\n');
+ }
+ });
}
- private static void metricWithRemoteCluster(Map<String, Collector.MetricFamilySamples> metrics, String cluster,
- String namespace, String name, String remoteCluster, double value) {
- addMetric(metrics, Map.of("cluster", cluster, "namespace", namespace, "remote_cluster", remoteCluster), name,
- value);
+ private static void writeReplicationStat(SimpleTextOutputStream stream, String cluster, String name,
+ List<AggregatedNamespaceStats> allNamespaceStats,
+ Function<AggregatedReplicationStats, Number> replStatsFunction) {
+ stream.write("# TYPE ").write(name).write(" gauge\n");
Review Comment:
how about placing it in `writeGuageType(metricName)`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on a diff in pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on code in PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#discussion_r912699155
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/NamespaceStatsAggregator.java:
##########
@@ -44,62 +51,65 @@
@Slf4j
public class NamespaceStatsAggregator {
- private static FastThreadLocal<AggregatedNamespaceStats> localNamespaceStats =
- new FastThreadLocal<AggregatedNamespaceStats>() {
+ private static final FastThreadLocal<AggregatedNamespaceStats> localNamespaceStats =
+ new FastThreadLocal<>() {
@Override
- protected AggregatedNamespaceStats initialValue() throws Exception {
+ protected AggregatedNamespaceStats initialValue() {
return new AggregatedNamespaceStats();
}
};
- private static FastThreadLocal<TopicStats> localTopicStats = new FastThreadLocal<TopicStats>() {
+ private static final FastThreadLocal<TopicStats> localTopicStats = new FastThreadLocal<>() {
@Override
- protected TopicStats initialValue() throws Exception {
+ protected TopicStats initialValue() {
return new TopicStats();
}
};
public static void generate(PulsarService pulsar, boolean includeTopicMetrics, boolean includeConsumerMetrics,
- boolean includeProducerMetrics, boolean splitTopicAndPartitionIndexLabel, SimpleTextOutputStream stream) {
+ boolean includeProducerMetrics, boolean splitTopicAndPartitionIndexLabel,
+ SimpleTextOutputStream stream) {
String cluster = pulsar.getConfiguration().getClusterName();
AggregatedNamespaceStats namespaceStats = localNamespaceStats.get();
- TopicStats.resetTypes();
TopicStats topicStats = localTopicStats.get();
-
- printDefaultBrokerStats(stream, cluster);
-
Optional<CompactorMXBean> compactorMXBean = getCompactorMXBean(pulsar);
LongAdder topicsCount = new LongAdder();
+ Map<String, ByteBuf> allMetrics = new HashMap<>();
Review Comment:
Not sure I understand the motivation here of creating a SimpleTextOutputStream and a ByteBuf for each metric. Using SimpleTextOutputStream per metric was my original PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
michaeljmarshall commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1235718390
Is there a reason we don't have this labeled for 2.8.5? I opened https://github.com/apache/pulsar/pull/17419 and would like to cherry pick it to 2.8.5. Cherry picking that will be much easier if this can get cherry picked first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on a diff in pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on code in PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#discussion_r898132375
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/metrics/PrometheusTextFormatUtil.java:
##########
@@ -141,29 +140,32 @@ private static void writeSum(Writer w, DataSketchesOpStatsLogger opStat, String
.append(Double.toString(opStat.getSum(success))).append('\n');
}
- public static void writeMetricsCollectedByPrometheusClient(Writer w, CollectorRegistry registry)
- throws IOException {
- Enumeration<MetricFamilySamples> metricFamilySamples = registry.metricFamilySamples();
- while (metricFamilySamples.hasMoreElements()) {
- MetricFamilySamples metricFamily = metricFamilySamples.nextElement();
-
- for (int i = 0; i < metricFamily.samples.size(); i++) {
- Sample sample = metricFamily.samples.get(i);
- w.write(sample.name);
- w.write('{');
- for (int j = 0; j < sample.labelNames.size(); j++) {
- if (j != 0) {
- w.write(", ");
+ public static void writeMetrics(SimpleTextOutputStream stream, Collection<MetricFamilySamples> familySamples) {
+ for (MetricFamilySamples familySample : familySamples) {
+ stream.write("# TYPE ");
+ stream.write(familySample.name);
+ stream.write(' ');
+ stream.write(familySample.type.name().toLowerCase());
+ stream.write('\n');
+ for (Collector.MetricFamilySamples.Sample sample : familySample.samples) {
+ stream.write(sample.name);
+ if (sample.labelNames.size() > 0) {
+ stream.write('{');
+ for (int i = 0; i < sample.labelNames.size(); ++i) {
+ stream.write(sample.labelNames.get(i));
+ stream.write("=\"");
+ stream.write(sample.labelValues.get(i));
+ stream.write("\",");
}
- w.write(sample.labelNames.get(j));
- w.write("=\"");
- w.write(sample.labelValues.get(j));
- w.write('"');
+ stream.write('}');
}
-
- w.write("} ");
- w.write(Collector.doubleToGoString(sample.value));
- w.write('\n');
+ stream.write(' ');
+ stream.write(sample.value);
Review Comment:
good spot, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] asafm commented on a diff in pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
asafm commented on code in PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#discussion_r913587012
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricsGenerator.java:
##########
@@ -181,22 +181,26 @@ private static ByteBuf generate0(PulsarService pulsar, boolean includeTopicMetri
//when write out 200MB data, MAX_COMPONENTS = 64 needn't mem_copy. see: CompositeByteBuf#consolidateIfNeeded()
ByteBuf buf = UnpooledByteBufAllocator.DEFAULT.compositeDirectBuffer(MAX_COMPONENTS);
boolean exceptionHappens = false;
+ //Used in namespace/topic and transaction aggregators as share metric names
+ PrometheusMetricStreams metricStream = new PrometheusMetricStreams();
Review Comment:
`metricStream` --> `metricStreams`
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricsGenerator.java:
##########
@@ -215,6 +219,7 @@ private static ByteBuf generate0(PulsarService pulsar, boolean includeTopicMetri
//if exception happens, release buffer
if (exceptionHappens) {
buf.release();
+ metricStream.releaseAll();
Review Comment:
The `if (exceptionHappens)` is there because the function returns that buffer hence you can't release a buffer that is about to be used. Only the caller knows when to release it. I would fix the comment to mention that.
In your case, the creator of the `ByteBuf` inside the metric streams is you, and it is not exposed outside, so it makes sense to release it regardless.
The original code is a bit risky since it can easily lead to a memory leak. It's best to allocate and free at the same method, but it's beyond the scope of this PR for sure.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricStreams.java:
##########
@@ -0,0 +1,74 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.stats.prometheus;
+
+import io.netty.buffer.ByteBufAllocator;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pulsar.common.util.SimpleTextOutputStream;
+
+/**
+ * Helper class to ensure that metrics of the same name are grouped together under the same TYPE header when written.
+ */
+public class PrometheusMetricStreams {
+ private final Map<String, SimpleTextOutputStream> metricStreamMap = new HashMap<>();
+
+ /**
+ * Write the given metric and sample value to the stream. Will write #TYPE header if metric not seen before.
+ * @param metricName name of the metric.
+ * @param value value of the sample
+ * @param labelsAndValuesArray varargs of label and label value
+ */
+ void writeSample(String metricName, Number value, String... labelsAndValuesArray) {
+ SimpleTextOutputStream stream = initGaugeType(metricName);
+ stream.write(metricName).write('{');
+ for (int i = 0; i < labelsAndValuesArray.length; i += 2) {
+ stream.write(labelsAndValuesArray[i]).write("=\"").write(labelsAndValuesArray[i + 1]).write('\"');
+ if (i + 2 != labelsAndValuesArray.length) {
+ stream.write(',');
+ }
+ }
+ stream.write("\"} ").write(value).write(' ').write(System.currentTimeMillis()).write('\n');
Review Comment:
In the original code, time was not written.
Exposition format mandates it optional, since it missing, the timestamp is assumed to be the scraping time.
The advantage of not writing it is saving time to download the response body to Prometheus, especially for a very large amount of topics.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricStreams.java:
##########
@@ -0,0 +1,74 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.stats.prometheus;
+
+import io.netty.buffer.ByteBufAllocator;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pulsar.common.util.SimpleTextOutputStream;
+
+/**
+ * Helper class to ensure that metrics of the same name are grouped together under the same TYPE header when written.
Review Comment:
I would also add: "Since those are the requirement of the Prometheus Exposition protocol"
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricsGenerator.java:
##########
@@ -181,22 +181,26 @@ private static ByteBuf generate0(PulsarService pulsar, boolean includeTopicMetri
//when write out 200MB data, MAX_COMPONENTS = 64 needn't mem_copy. see: CompositeByteBuf#consolidateIfNeeded()
ByteBuf buf = UnpooledByteBufAllocator.DEFAULT.compositeDirectBuffer(MAX_COMPONENTS);
boolean exceptionHappens = false;
+ //Used in namespace/topic and transaction aggregators as share metric names
Review Comment:
Do they share metrics? how? Each has its own prefix, no?
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricStreams.java:
##########
@@ -0,0 +1,74 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.stats.prometheus;
+
+import io.netty.buffer.ByteBufAllocator;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pulsar.common.util.SimpleTextOutputStream;
+
+/**
+ * Helper class to ensure that metrics of the same name are grouped together under the same TYPE header when written.
+ */
+public class PrometheusMetricStreams {
+ private final Map<String, SimpleTextOutputStream> metricStreamMap = new HashMap<>();
+
+ /**
+ * Write the given metric and sample value to the stream. Will write #TYPE header if metric not seen before.
+ * @param metricName name of the metric.
+ * @param value value of the sample
+ * @param labelsAndValuesArray varargs of label and label value
+ */
+ void writeSample(String metricName, Number value, String... labelsAndValuesArray) {
+ SimpleTextOutputStream stream = initGaugeType(metricName);
+ stream.write(metricName).write('{');
+ for (int i = 0; i < labelsAndValuesArray.length; i += 2) {
+ stream.write(labelsAndValuesArray[i]).write("=\"").write(labelsAndValuesArray[i + 1]).write('\"');
+ if (i + 2 != labelsAndValuesArray.length) {
+ stream.write(',');
+ }
+ }
+ stream.write("\"} ").write(value).write(' ').write(System.currentTimeMillis()).write('\n');
+ }
+
+ /**
+ * Flush all the stored metrics to the supplied stream.
+ * @param stream the stream to write to.
+ */
+ void flushAllToStream(SimpleTextOutputStream stream) {
+ metricStreamMap.values().forEach(s -> stream.write(s.getBuffer()));
+ releaseAll();
Review Comment:
I would call release once in finally
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
eolivelli commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1217672943
there is a checkstyle violation:
> [INFO] There is 1 error reported by Checkstyle 8.37 with ../buildtools/src/main/resources/pulsar/checkstyle.xml ruleset.
> Error: src/main/java/org/apache/pulsar/broker/stats/prometheus/NamespaceStatsAggregator.java:[372] (regexp) RegexpSingleline: Trailing whitespace
> [INFO] ------------------------------------------------------------------------
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1238999863
@michaeljmarshall - DataDog seems to expect the prometheus metrics to be grouped together by name, this is usually under a `#TYPE` heading. If the metrics are not grouped then DataDog ignores any under a `#TYPE` header that do not have the name in that header.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1245415676
@mattisonchao created https://github.com/apache/pulsar/pull/17618 to merge into branch-2.9.
Is similar needed for branch-2.10?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on a diff in pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on code in PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#discussion_r936461737
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricStreams.java:
##########
@@ -0,0 +1,75 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.stats.prometheus;
+
+import io.netty.buffer.ByteBufAllocator;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pulsar.common.util.SimpleTextOutputStream;
+
+/**
+ * Helper class to ensure that metrics of the same name are grouped together under the same TYPE header when written.
+ * Those are the requirements of the
+ * <a href="https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md#grouping-and-sorting">Prometheus Exposition Format</a>.
+ */
+public class PrometheusMetricStreams {
+ private final Map<String, SimpleTextOutputStream> metricStreamMap = new HashMap<>();
+
+ /**
+ * Write the given metric and sample value to the stream. Will write #TYPE header if metric not seen before.
+ * @param metricName name of the metric.
+ * @param value value of the sample
+ * @param labelsAndValuesArray varargs of label and label value
+ */
+ void writeSample(String metricName, Number value, String... labelsAndValuesArray) {
+ SimpleTextOutputStream stream = initGaugeType(metricName);
+ stream.write(metricName).write('{');
+ for (int i = 0; i < labelsAndValuesArray.length; i += 2) {
+ stream.write(labelsAndValuesArray[i]).write("=\"").write(labelsAndValuesArray[i + 1]).write('\"');
+ if (i + 2 != labelsAndValuesArray.length) {
+ stream.write(',');
+ }
+ }
+ stream.write("\"} ").write(value).write(' ').write(System.currentTimeMillis()).write('\n');
Review Comment:
good spot, not needed here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] asafm commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
asafm commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1186459428
@merlimat Sorry to nag again - could you please take a look at the comments I mentioned you at? I want to be certain here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] nahguam commented on a diff in pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
nahguam commented on code in PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#discussion_r908195822
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java:
##########
@@ -1501,6 +1503,59 @@ private void compareCompactionStateCount(List<Metric> cm, double count) {
assertEquals(cm.get(0).value, count);
}
+ @Test
+ public void testMetricsGroupedByTypeDefinitions() throws Exception {
+ Producer<byte[]> p1 = pulsarClient.newProducer().topic("persistent://my-property/use/my-ns/my-topic1").create();
+ Producer<byte[]> p2 = pulsarClient.newProducer().topic("persistent://my-property/use/my-ns/my-topic2").create();
+ for (int i = 0; i < 10; i++) {
+ String message = "my-message-" + i;
+ p1.send(message.getBytes());
+ p2.send(message.getBytes());
+ }
+
+ ByteArrayOutputStream statsOut = new ByteArrayOutputStream();
+ PrometheusMetricsGenerator.generate(pulsar, false, false, false, statsOut);
+ String metricsStr = statsOut.toString();
+
+ Pattern typePattern = Pattern.compile("^#\\s+TYPE\\s+(\\w+)\\s+(\\w+)");
+ Pattern metricNamePattern = Pattern.compile("^(\\w+)\\{.+");
+
+ AtomicReference<String> currentMetric = new AtomicReference<>();
+ Splitter.on("\n").split(metricsStr).forEach(line -> {
+ if (line.isEmpty()) {
+ return;
+ }
+ if (line.startsWith("#")) {
+ // Get the current type definition
+ Matcher typeMatcher = typePattern.matcher(line);
+ checkArgument(typeMatcher.matches());
+ String metricName = typeMatcher.group(1);
+ currentMetric.set(metricName);
+ } else {
+ Matcher metricMatcher = metricNamePattern.matcher(line);
+ checkArgument(metricMatcher.matches());
+ String metricName = metricMatcher.group(1);
+
+ if (metricName.endsWith("_bucket")) {
+ metricName = metricName.substring(0, metricName.indexOf("_bucket"));
+ } else if (metricName.endsWith("_count") && !currentMetric.get().endsWith("_count")) {
+ metricName = metricName.substring(0, metricName.indexOf("_count"));
+ } else if (metricName.endsWith("_sum") && !currentMetric.get().endsWith("_sum")) {
+ metricName = metricName.substring(0, metricName.indexOf("_sum"));
+ }
+
+ if (!metricName.equals(currentMetric.get())) {
+ System.out.println(metricsStr);
Review Comment:
Is this sysout needed, or could this be a log trace/debug?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by "Technoboy- (via GitHub)" <gi...@apache.org>.
Technoboy- commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1422084899
#17419 relies on this, so cherry-picked to branch-2.11
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
liangyepianzhou commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1350219764
>@mattisonchao created https://github.com/apache/pulsar/pull/17618 to merge into branch-2.9.
Is similar needed for branch-2.10?
@marksilcox This PR does need to cherry-pick to branch-2.10.
I am trying to cherry-pick it to branch-2.10, but there are a lot of conflicts.
Could you please open a PR to cherry-pick this PR to branch-2.10 (to avoid including bugs)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1132614973
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on a diff in pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on code in PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#discussion_r872495351
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/TopicStats.java:
##########
@@ -106,269 +112,270 @@ static void resetTypes() {
metricWithTypeDefinition.clear();
}
- static void printTopicStats(SimpleTextOutputStream stream, String cluster, String namespace, String topic,
+ static void printTopicStats(Map<String, List<String>> metrics, String cluster, String namespace, String topic,
Review Comment:
updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1165434030
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1162755427
@marksilcox Please provide a correct documentation label for your PR.
Instructions see [Pulsar Documentation Label Guide](https://docs.google.com/document/d/1Qw7LHQdXWBW9t2-r-A7QdFDBwmZh6ytB4guwMoXHqc0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] asafm commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
asafm commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1351721380
Just want to remind everyone that when a new PR was created to cherry-pick it from master to 2.9, a memory leak was introduced since the 2.9 metrics implementation is different a bit from the master branch. So we need to double-check that. See [memory leak fixed](https://github.com/apache/pulsar/pull/17852)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] asafm commented on a diff in pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
asafm commented on code in PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#discussion_r936694269
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricStreamsTest.java:
##########
@@ -0,0 +1,85 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.stats.prometheus;
+
+import static org.testng.Assert.assertTrue;
+import io.netty.buffer.ByteBuf;
+import io.netty.buffer.ByteBufAllocator;
+import java.io.ByteArrayOutputStream;
+import java.nio.charset.StandardCharsets;
+import org.apache.pulsar.common.util.SimpleTextOutputStream;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+@Test(groups = "broker")
+public class PrometheusMetricStreamsTest {
Review Comment:
💯
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jason918 commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
Jason918 commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1233665122
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1234456773
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
michaeljmarshall commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1238862774
> I don't think we should hurry and port too many "improvements" like this to branch-2.10
@eolivelli - I think it depends on whether this is an improvement or a bug fix. The PR's description cites Prometheus documentation here:
> Prometheus docs states "All lines for a given metric must be provided as one single group" - https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md#grouping-and-sorting
When I first read that line, I interpreted it to mean lines with the same metric name, e.g. `http_requests_total`, must be grouped together, not that all metrics of the same type must be grouped together. However, it might be an ambiguous rule. At the very least, it seems like Prometheus does not require metrics of the same type to be grouped but DataDog does.
@marksilcox - I am not familiar with DataDog. What is the consequence of these metrics being ungrouped for DataDog?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on a diff in pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on code in PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#discussion_r912680425
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/metrics/PrometheusTextFormatUtil.java:
##########
@@ -141,29 +140,32 @@ private static void writeSum(Writer w, DataSketchesOpStatsLogger opStat, String
.append(Double.toString(opStat.getSum(success))).append('\n');
}
- public static void writeMetricsCollectedByPrometheusClient(Writer w, CollectorRegistry registry)
- throws IOException {
- Enumeration<MetricFamilySamples> metricFamilySamples = registry.metricFamilySamples();
- while (metricFamilySamples.hasMoreElements()) {
- MetricFamilySamples metricFamily = metricFamilySamples.nextElement();
-
- for (int i = 0; i < metricFamily.samples.size(); i++) {
- Sample sample = metricFamily.samples.get(i);
- w.write(sample.name);
- w.write('{');
- for (int j = 0; j < sample.labelNames.size(); j++) {
- if (j != 0) {
- w.write(", ");
+ public static void writeMetrics(SimpleTextOutputStream stream, Collection<MetricFamilySamples> familySamples) {
+ for (MetricFamilySamples familySample : familySamples) {
+ stream.write("# TYPE ");
Review Comment:
This method is not used anywhere since an earlier refactor - removing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] asafm commented on a diff in pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
asafm commented on code in PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#discussion_r913070560
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/NamespaceStatsAggregator.java:
##########
@@ -44,62 +51,65 @@
@Slf4j
public class NamespaceStatsAggregator {
- private static FastThreadLocal<AggregatedNamespaceStats> localNamespaceStats =
- new FastThreadLocal<AggregatedNamespaceStats>() {
+ private static final FastThreadLocal<AggregatedNamespaceStats> localNamespaceStats =
+ new FastThreadLocal<>() {
@Override
- protected AggregatedNamespaceStats initialValue() throws Exception {
+ protected AggregatedNamespaceStats initialValue() {
return new AggregatedNamespaceStats();
}
};
- private static FastThreadLocal<TopicStats> localTopicStats = new FastThreadLocal<TopicStats>() {
+ private static final FastThreadLocal<TopicStats> localTopicStats = new FastThreadLocal<>() {
@Override
- protected TopicStats initialValue() throws Exception {
+ protected TopicStats initialValue() {
return new TopicStats();
}
};
public static void generate(PulsarService pulsar, boolean includeTopicMetrics, boolean includeConsumerMetrics,
- boolean includeProducerMetrics, boolean splitTopicAndPartitionIndexLabel, SimpleTextOutputStream stream) {
+ boolean includeProducerMetrics, boolean splitTopicAndPartitionIndexLabel,
+ SimpleTextOutputStream stream) {
String cluster = pulsar.getConfiguration().getClusterName();
AggregatedNamespaceStats namespaceStats = localNamespaceStats.get();
- TopicStats.resetTypes();
TopicStats topicStats = localTopicStats.get();
-
- printDefaultBrokerStats(stream, cluster);
-
Optional<CompactorMXBean> compactorMXBean = getCompactorMXBean(pulsar);
LongAdder topicsCount = new LongAdder();
+ Map<String, ByteBuf> allMetrics = new HashMap<>();
Review Comment:
The idea is to have a separate `ByteBuf` per metric, right?
`SimpleTextOutputStream` is just a wrapper around an existing `ByteBuf`, which gives serialization functionality of writing int, long, ByteBuf, etc into a ByteBuf.
So instead of copy-pasting the functionality from `SimpleTextOutputStream` into `NamespaceStatsAggregator`, and using the copied functionality to write into the metric `ByteBuf`, you wrap that `ByteBuf` with `SimpleTextOutputStream` to gain that functionality.
So the change is: `Map<String, ByteBuf>` to `Map<String, SimpleTextOutputStream>`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1164645963
@asafm I have now updated following the pattern you suggest. I'm not totally familiar with `FastThreadLocal` but I think I have minimised the object creation where possible. Would be grateful for some further pointers if this could be improved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1174694798
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] asafm commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
asafm commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1207908433
@eolivelli Do you want to re-review your requested changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] asafm commented on a diff in pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
asafm commented on code in PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#discussion_r938020746
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/TopicStats.java:
##########
@@ -102,380 +104,401 @@ public void reset() {
compactionLatencyBuckets.reset();
}
- static void resetTypes() {
- metricWithTypeDefinition.clear();
- }
-
- static void printTopicStats(SimpleTextOutputStream stream, String cluster, String namespace, String topic,
- TopicStats stats, Optional<CompactorMXBean> compactorMXBean,
- boolean splitTopicAndPartitionIndexLabel) {
- metric(stream, cluster, namespace, topic, "pulsar_subscriptions_count", stats.subscriptionsCount,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_producers_count", stats.producersCount,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_consumers_count", stats.consumersCount,
- splitTopicAndPartitionIndexLabel);
-
- metric(stream, cluster, namespace, topic, "pulsar_rate_in", stats.rateIn,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_rate_out", stats.rateOut,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_throughput_in", stats.throughputIn,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_throughput_out", stats.throughputOut,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_average_msg_size", stats.averageMsgSize,
- splitTopicAndPartitionIndexLabel);
-
- metric(stream, cluster, namespace, topic, "pulsar_storage_size", stats.managedLedgerStats.storageSize,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_logical_size",
- stats.managedLedgerStats.storageLogicalSize, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_msg_backlog", stats.msgBacklog,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_backlog_size",
- stats.managedLedgerStats.backlogSize, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_publish_rate_limit_times", stats.publishRateLimitedTimes,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_offloaded_size", stats.managedLedgerStats
- .offloadedStorageUsed, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_backlog_quota_limit", stats.backlogQuotaLimit,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_backlog_quota_limit_time",
- stats.backlogQuotaLimitTime, splitTopicAndPartitionIndexLabel);
-
- long[] latencyBuckets = stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets();
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_le_0_5", latencyBuckets[0],
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_le_1", latencyBuckets[1],
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_le_5", latencyBuckets[2],
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_le_10", latencyBuckets[3],
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_le_20", latencyBuckets[4],
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_le_50", latencyBuckets[5],
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_le_100", latencyBuckets[6],
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_le_200", latencyBuckets[7],
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_le_1000", latencyBuckets[8],
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_overflow", latencyBuckets[9],
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_count",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getCount(), splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_sum",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getSum(), splitTopicAndPartitionIndexLabel);
-
- long[] ledgerWriteLatencyBuckets = stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets();
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_0_5",
- ledgerWriteLatencyBuckets[0], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_1",
- ledgerWriteLatencyBuckets[1], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_5",
- ledgerWriteLatencyBuckets[2], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_10",
- ledgerWriteLatencyBuckets[3], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_20",
- ledgerWriteLatencyBuckets[4], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_50",
- ledgerWriteLatencyBuckets[5], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_100",
- ledgerWriteLatencyBuckets[6], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_200",
- ledgerWriteLatencyBuckets[7], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_1000",
- ledgerWriteLatencyBuckets[8], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_overflow",
- ledgerWriteLatencyBuckets[9], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_count",
+ public static void printTopicStats(Map<String, ByteBuf> allMetrics, TopicStats stats,
+ Optional<CompactorMXBean> compactorMXBean, String cluster, String namespace,
+ String name, boolean splitTopicAndPartitionIndexLabel) {
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_subscriptions_count", stats.subscriptionsCount,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_producers_count", stats.producersCount,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_consumers_count", stats.consumersCount,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_rate_in", stats.rateIn,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_rate_out", stats.rateOut,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_throughput_in", stats.throughputIn,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_throughput_out", stats.throughputOut,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_average_msg_size", stats.averageMsgSize,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_storage_size", stats.managedLedgerStats.storageSize,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_storage_logical_size",
+ stats.managedLedgerStats.storageLogicalSize, cluster, namespace, name,
+ splitTopicAndPartitionIndexLabel);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_msg_backlog", stats.msgBacklog,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_backlog_size", stats.managedLedgerStats.backlogSize,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_publish_rate_limit_times", stats.publishRateLimitedTimes,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_offloaded_size", stats.managedLedgerStats
+ .offloadedStorageUsed, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_backlog_quota_limit", stats.backlogQuotaLimit,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_backlog_quota_limit_time", stats.backlogQuotaLimitTime,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_0_5",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[0],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_1",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[1],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_5",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[2],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_10",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[3],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_20",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[4],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_50",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[5],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_100",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[6],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_200",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[7],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_1000",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[8],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_overflow",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[9],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_count",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getCount(),
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_sum",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getSum(), cluster, namespace, name,
+ splitTopicAndPartitionIndexLabel);
+
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_0_5",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[0],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_1",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[1],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_5",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[2],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_10",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[3],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_20",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[4],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_50",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[5],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_100",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[6],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_200",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[7],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_1000",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[8],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_overflow",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[9],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_count",
stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getCount(),
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_sum",
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_sum",
stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getSum(),
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+ writeMetric(allMetrics, "pulsar_entry_size_le_128",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[0],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_entry_size_le_512",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[1],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_entry_size_le_1_kb",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[2],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_entry_size_le_2_kb",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[3],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_entry_size_le_4_kb",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[4],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_entry_size_le_16_kb",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[5],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_entry_size_le_100_kb",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[6],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_entry_size_le_1_mb",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[7],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_entry_size_le_overflow",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[8],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_entry_size_count", stats.managedLedgerStats.entrySizeBuckets.getCount(),
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_entry_size_sum", stats.managedLedgerStats.entrySizeBuckets.getSum(),
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+ writeProducerStat(allMetrics, "pulsar_producer_msg_rate_in", stats,
+ p -> p.msgRateIn, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeProducerStat(allMetrics, "pulsar_producer_msg_throughput_in", stats,
+ p -> p.msgThroughputIn, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeProducerStat(allMetrics, "pulsar_producer_msg_average_Size", stats,
+ p -> p.averageMsgSize, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_back_log", stats, s -> s.msgBacklog,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_back_log_no_delayed",
+ stats, s -> s.msgBacklogNoDelayed, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_delayed",
+ stats, s -> s.msgDelayed, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_msg_rate_redeliver",
+ stats, s -> s.msgRateRedeliver, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_unacked_messages",
+ stats, s -> s.unackedMessages, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_blocked_on_unacked_messages",
+ stats, s -> s.blockedSubscriptionOnUnackedMsgs ? 1 : 0, cluster, namespace, name,
+ splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_msg_rate_out",
+ stats, s -> s.msgRateOut, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_msg_ack_rate",
+ stats, s -> s.messageAckRate, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_msg_throughput_out",
+ stats, s -> s.msgThroughputOut, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_out_bytes_total",
+ stats, s -> s.bytesOutCounter, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_out_messages_total",
+ stats, s -> s.msgOutCounter, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_last_expire_timestamp",
+ stats, s -> s.lastExpireTimestamp, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_last_acked_timestamp",
+ stats, s -> s.lastAckedTimestamp, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_last_consumed_flow_timestamp",
+ stats, s -> s.lastConsumedFlowTimestamp, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_last_consumed_timestamp",
+ stats, s -> s.lastConsumedTimestamp, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_last_mark_delete_advanced_timestamp",
+ stats, s -> s.lastMarkDeleteAdvancedTimestamp, cluster, namespace, name,
+ splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_msg_rate_expired",
+ stats, s -> s.msgRateExpired, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_total_msg_expired",
+ stats, s -> s.totalMsgExpired, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_msg_drop_rate",
+ stats, s -> s.msgDropRate, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_consumers_count",
+ stats, s -> s.consumersCount, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+ writeConsumerStat(allMetrics, "pulsar_consumer_msg_rate_redeliver", stats, c -> c.msgRateRedeliver,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeConsumerStat(allMetrics, "pulsar_consumer_unacked_messages", stats, c -> c.unackedMessages,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeConsumerStat(allMetrics, "pulsar_consumer_blocked_on_unacked_messages",
+ stats, c -> c.blockedSubscriptionOnUnackedMsgs ? 1 : 0,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeConsumerStat(allMetrics, "pulsar_consumer_msg_rate_out", stats, c -> c.msgRateOut,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+ writeConsumerStat(allMetrics, "pulsar_consumer_msg_ack_rate", stats, c -> c.msgAckRate,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+ writeConsumerStat(allMetrics, "pulsar_consumer_msg_throughput_out", stats, c -> c.msgThroughputOut,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeConsumerStat(allMetrics, "pulsar_consumer_available_permits", stats, c -> c.availablePermits,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeConsumerStat(allMetrics, "pulsar_out_bytes_total", stats, c -> c.bytesOutCounter,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeConsumerStat(allMetrics, "pulsar_out_messages_total", stats, c -> c.msgOutCounter,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+ writeReplicationStat(allMetrics, "pulsar_replication_rate_in", stats, r -> r.msgRateIn,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(allMetrics, "pulsar_replication_rate_out", stats, r -> r.msgRateOut,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(allMetrics, "pulsar_replication_throughput_in", stats, r -> r.msgThroughputIn,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(allMetrics, "pulsar_replication_throughput_out", stats, r -> r.msgThroughputOut,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(allMetrics, "pulsar_replication_backlog", stats, r -> r.replicationBacklog,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(allMetrics, "pulsar_replication_connected_count", stats, r -> r.connectedCount,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(allMetrics, "pulsar_replication_rate_expired", stats, r -> r.msgRateExpired,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(allMetrics, "pulsar_replication_delay_in_seconds", stats,
+ r -> r.replicationDelayInSeconds, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+ writeMetric(allMetrics, "pulsar_in_bytes_total", stats.bytesInCounter, cluster, namespace, name,
+ splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_in_messages_total", stats.msgInCounter, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- long[] entrySizeBuckets = stats.managedLedgerStats.entrySizeBuckets.getBuckets();
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_le_128", entrySizeBuckets[0],
+ // Compaction
+
+ writeCompactionStat(allMetrics, "pulsar_compaction_removed_event_count", stats.compactionRemovedEventCount,
+ compactorMXBean, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(allMetrics, "pulsar_compaction_succeed_count", stats.compactionSucceedCount,
+ compactorMXBean, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(allMetrics, "pulsar_compaction_failed_count", stats.compactionFailedCount, compactorMXBean,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(allMetrics, "pulsar_compaction_duration_time_in_mills", stats.compactionDurationTimeInMills,
+ compactorMXBean, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(allMetrics, "pulsar_compaction_read_throughput", stats.compactionReadThroughput,
+ compactorMXBean, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(allMetrics, "pulsar_compaction_write_throughput", stats.compactionWriteThroughput,
+ compactorMXBean, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(allMetrics, "pulsar_compaction_compacted_entries_count",
+ stats.compactionCompactedEntriesCount, compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_le_512", entrySizeBuckets[1],
+ writeCompactionStat(allMetrics, "pulsar_compaction_compacted_entries_size",
+ stats.compactionCompactedEntriesSize, compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_le_1_kb", entrySizeBuckets[2],
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_le_0_5",
+ stats.compactionLatencyBuckets.getBuckets()[0], compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_le_2_kb", entrySizeBuckets[3],
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_le_1",
+ stats.compactionLatencyBuckets.getBuckets()[1], compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_le_4_kb", entrySizeBuckets[4],
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_le_5",
+ stats.compactionLatencyBuckets.getBuckets()[2], compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_le_16_kb", entrySizeBuckets[5],
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_le_10",
+ stats.compactionLatencyBuckets.getBuckets()[3], compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_le_100_kb", entrySizeBuckets[6],
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_le_20",
+ stats.compactionLatencyBuckets.getBuckets()[4], compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_le_1_mb", entrySizeBuckets[7],
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_le_50",
+ stats.compactionLatencyBuckets.getBuckets()[5], compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_le_overflow", entrySizeBuckets[8],
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_le_100",
+ stats.compactionLatencyBuckets.getBuckets()[6], compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_count",
- stats.managedLedgerStats.entrySizeBuckets.getCount(), splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_sum",
- stats.managedLedgerStats.entrySizeBuckets.getSum(), splitTopicAndPartitionIndexLabel);
-
- stats.producerStats.forEach((p, producerStats) -> {
- metric(stream, cluster, namespace, topic, p, producerStats.producerId, "pulsar_producer_msg_rate_in",
- producerStats.msgRateIn, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, p, producerStats.producerId, "pulsar_producer_msg_throughput_in",
- producerStats.msgThroughputIn, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, p, producerStats.producerId, "pulsar_producer_msg_average_Size",
- producerStats.averageMsgSize, splitTopicAndPartitionIndexLabel);
- });
-
- stats.subscriptionStats.forEach((n, subsStats) -> {
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_back_log",
- subsStats.msgBacklog, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_back_log_no_delayed",
- subsStats.msgBacklogNoDelayed, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_delayed",
- subsStats.msgDelayed, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_msg_rate_redeliver",
- subsStats.msgRateRedeliver, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_unacked_messages",
- subsStats.unackedMessages, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_blocked_on_unacked_messages",
- subsStats.blockedSubscriptionOnUnackedMsgs ? 1 : 0, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_msg_rate_out",
- subsStats.msgRateOut, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_msg_ack_rate",
- subsStats.messageAckRate, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_msg_throughput_out",
- subsStats.msgThroughputOut, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_out_bytes_total",
- subsStats.bytesOutCounter, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_out_messages_total",
- subsStats.msgOutCounter, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_last_expire_timestamp",
- subsStats.lastExpireTimestamp, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_last_acked_timestamp",
- subsStats.lastAckedTimestamp, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_last_consumed_flow_timestamp",
- subsStats.lastConsumedFlowTimestamp, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_last_consumed_timestamp",
- subsStats.lastConsumedTimestamp, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_last_mark_delete_advanced_timestamp",
- subsStats.lastMarkDeleteAdvancedTimestamp, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_msg_rate_expired",
- subsStats.msgRateExpired, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_total_msg_expired",
- subsStats.totalMsgExpired, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_msg_drop_rate",
- subsStats.msgDropRate, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_consumers_count",
- subsStats.consumersCount, splitTopicAndPartitionIndexLabel);
- subsStats.consumerStat.forEach((c, consumerStats) -> {
- metric(stream, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_msg_rate_redeliver", consumerStats.msgRateRedeliver,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_unacked_messages", consumerStats.unackedMessages,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_blocked_on_unacked_messages",
- consumerStats.blockedSubscriptionOnUnackedMsgs ? 1 : 0,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_msg_rate_out", consumerStats.msgRateOut,
- splitTopicAndPartitionIndexLabel);
-
- metric(stream, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_msg_ack_rate", consumerStats.msgAckRate,
- splitTopicAndPartitionIndexLabel);
-
- metric(stream, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_msg_throughput_out", consumerStats.msgThroughputOut,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_available_permits", consumerStats.availablePermits,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_out_bytes_total", consumerStats.bytesOutCounter,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_out_messages_total", consumerStats.msgOutCounter,
- splitTopicAndPartitionIndexLabel);
- });
- });
-
- if (!stats.replicationStats.isEmpty()) {
- stats.replicationStats.forEach((remoteCluster, replStats) -> {
- metricWithRemoteCluster(stream, cluster, namespace, topic, "pulsar_replication_rate_in", remoteCluster,
- replStats.msgRateIn, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(stream, cluster, namespace, topic, "pulsar_replication_rate_out", remoteCluster,
- replStats.msgRateOut, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(stream, cluster, namespace, topic, "pulsar_replication_throughput_in",
- remoteCluster, replStats.msgThroughputIn, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(stream, cluster, namespace, topic, "pulsar_replication_throughput_out",
- remoteCluster, replStats.msgThroughputOut, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(stream, cluster, namespace, topic, "pulsar_replication_backlog", remoteCluster,
- replStats.replicationBacklog, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(stream, cluster, namespace, topic, "pulsar_replication_connected_count",
- remoteCluster, replStats.connectedCount, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(stream, cluster, namespace, topic, "pulsar_replication_rate_expired",
- remoteCluster, replStats.msgRateExpired, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(stream, cluster, namespace, topic, "pulsar_replication_delay_in_seconds",
- remoteCluster, replStats.replicationDelayInSeconds, splitTopicAndPartitionIndexLabel);
- });
- }
-
- metric(stream, cluster, namespace, topic, "pulsar_in_bytes_total", stats.bytesInCounter,
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_le_200",
+ stats.compactionLatencyBuckets.getBuckets()[7], compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_in_messages_total", stats.msgInCounter,
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_le_1000",
+ stats.compactionLatencyBuckets.getBuckets()[8], compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
-
- // Compaction
- boolean hasCompaction = compactorMXBean.flatMap(mxBean -> mxBean.getCompactionRecordForTopic(topic))
- .map(__ -> true).orElse(false);
- if (hasCompaction) {
- metric(stream, cluster, namespace, topic, "pulsar_compaction_removed_event_count",
- stats.compactionRemovedEventCount, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_succeed_count",
- stats.compactionSucceedCount, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_failed_count",
- stats.compactionFailedCount, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_duration_time_in_mills",
- stats.compactionDurationTimeInMills, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_read_throughput",
- stats.compactionReadThroughput, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_write_throughput",
- stats.compactionWriteThroughput, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_compacted_entries_count",
- stats.compactionCompactedEntriesCount, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_compacted_entries_size",
- stats.compactionCompactedEntriesSize, splitTopicAndPartitionIndexLabel);
- long[] compactionLatencyBuckets = stats.compactionLatencyBuckets.getBuckets();
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_le_0_5",
- compactionLatencyBuckets[0], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_le_1",
- compactionLatencyBuckets[1], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_le_5",
- compactionLatencyBuckets[2], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_le_10",
- compactionLatencyBuckets[3], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_le_20",
- compactionLatencyBuckets[4], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_le_50",
- compactionLatencyBuckets[5], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_le_100",
- compactionLatencyBuckets[6], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_le_200",
- compactionLatencyBuckets[7], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_le_1000",
- compactionLatencyBuckets[8], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_overflow",
- compactionLatencyBuckets[9], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_sum",
- stats.compactionLatencyBuckets.getSum(), splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_count",
- stats.compactionLatencyBuckets.getCount(), splitTopicAndPartitionIndexLabel);
- }
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_overflow",
+ stats.compactionLatencyBuckets.getBuckets()[9], compactorMXBean, cluster, namespace, name,
+ splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_sum", stats.compactionLatencyBuckets.getSum(),
+ compactorMXBean, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_count", stats.compactionLatencyBuckets.getCount(),
+ compactorMXBean, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
}
- static void metricType(SimpleTextOutputStream stream, String name) {
-
- if (!metricWithTypeDefinition.containsKey(name)) {
- metricWithTypeDefinition.put(name, "gauge");
- stream.write("# TYPE ").write(name).write(" gauge\n");
- }
-
+ private static void writeMetric(Map<String, ByteBuf> allMetrics, String metricName, Number value, String cluster,
+ String namespace, String topic, boolean splitTopicAndPartitionIndexLabel) {
+ ByteBuf buffer = writeGaugeType(allMetrics, metricName);
+ writeTopicSample(buffer, metricName, value, cluster, namespace, topic, splitTopicAndPartitionIndexLabel);
}
- private static void metric(SimpleTextOutputStream stream, String cluster, String namespace, String topic,
- String name, double value, boolean splitTopicAndPartitionIndexLabel) {
- metricType(stream, name);
- appendRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel).write("\"} ");
- stream.write(value);
- appendEndings(stream);
+ private static void writeProducerStat(Map<String, ByteBuf> allMetrics, String metricName, TopicStats topicStats,
+ Function<AggregatedProducerStats, Number> valueFunction,
+ String cluster, String namespace, String topic,
+ boolean splitTopicAndPartitionIndexLabel) {
+ ByteBuf buffer = writeGaugeType(allMetrics, metricName);
+ topicStats.producerStats.forEach((p, producerStats) ->
+ writeTopicSample(buffer, metricName, valueFunction.apply(producerStats), cluster, namespace, topic,
+ splitTopicAndPartitionIndexLabel, "producer_name", p, "producer_id",
+ String.valueOf(producerStats.producerId)));
}
- private static void metric(SimpleTextOutputStream stream, String cluster, String namespace, String topic,
- String subscription, String name, long value, boolean splitTopicAndPartitionIndexLabel) {
- metricType(stream, name);
- appendRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel)
- .write("\",subscription=\"").write(subscription).write("\"} ");
- stream.write(value);
- appendEndings(stream);
+ private static void writeSubscriptionStat(Map<String, ByteBuf> allMetrics, String metricName, TopicStats topicStats,
+ Function<AggregatedSubscriptionStats, Number> valueFunction,
+ String cluster, String namespace, String topic,
+ boolean splitTopicAndPartitionIndexLabel) {
+ ByteBuf buffer = writeGaugeType(allMetrics, metricName);
+ topicStats.subscriptionStats.forEach((s, subStats) ->
+ writeTopicSample(buffer, metricName, valueFunction.apply(subStats), cluster, namespace, topic,
+ splitTopicAndPartitionIndexLabel, "subscription", s));
}
- private static void metric(SimpleTextOutputStream stream, String cluster, String namespace, String topic,
- String producerName, long produceId, String name, double value, boolean splitTopicAndPartitionIndexLabel) {
- metricType(stream, name);
- appendRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel)
- .write("\",producer_name=\"").write(producerName)
- .write("\",producer_id=\"").write(produceId).write("\"} ");
- stream.write(value);
- appendEndings(stream);
+ private static void writeConsumerStat(Map<String, ByteBuf> allMetrics, String metricName, TopicStats topicStats,
+ Function<AggregatedConsumerStats, Number> valueFunction,
+ String cluster, String namespace, String topic,
+ boolean splitTopicAndPartitionIndexLabel) {
+ ByteBuf buffer = writeGaugeType(allMetrics, metricName);
+ topicStats.subscriptionStats.forEach((s, subStats) ->
+ subStats.consumerStat.forEach((c, conStats) ->
+ writeTopicSample(buffer, metricName, valueFunction.apply(conStats), cluster, namespace, topic,
+ splitTopicAndPartitionIndexLabel, "subscription", s, "consumer_name", c.consumerName(),
+ "consumer_id", String.valueOf(c.consumerId()))
+ ));
}
- private static void metric(SimpleTextOutputStream stream, String cluster, String namespace, String topic,
- String subscription, String name, double value, boolean splitTopicAndPartitionIndexLabel) {
- metricType(stream, name);
- appendRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel)
- .write("\",subscription=\"").write(subscription).write("\"} ");
- stream.write(value);
- appendEndings(stream);
- }
- private static void metric(SimpleTextOutputStream stream, String cluster, String namespace, String topic,
- String subscription, String consumerName, long consumerId, String name, long value,
- boolean splitTopicAndPartitionIndexLabel) {
- metricType(stream, name);
- appendRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel)
- .write("\",subscription=\"").write(subscription)
- .write("\",consumer_name=\"").write(consumerName).write("\",consumer_id=\"").write(consumerId)
- .write("\"} ");
- stream.write(value);
- appendEndings(stream);
+ private static void writeReplicationStat(Map<String, ByteBuf> allMetrics, String metricName, TopicStats topicStats,
+ Function<AggregatedReplicationStats, Number> valueFunction,
+ String cluster, String namespace, String topic,
+ boolean splitTopicAndPartitionIndexLabel) {
+ if (!topicStats.replicationStats.isEmpty()) {
+ ByteBuf buffer = writeGaugeType(allMetrics, metricName);
+ topicStats.replicationStats.forEach((remoteCluster, replStats) ->
+ writeTopicSample(buffer, metricName, valueFunction.apply(replStats), cluster, namespace, topic,
+ splitTopicAndPartitionIndexLabel, "remote_cluster", remoteCluster)
+ );
+ }
}
- private static void metric(SimpleTextOutputStream stream, String cluster, String namespace, String topic,
- String subscription, String consumerName, long consumerId, String name, double value,
- boolean splitTopicAndPartitionIndexLabel) {
- metricType(stream, name);
- appendRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel)
- .write("\",subscription=\"").write(subscription)
- .write("\",consumer_name=\"").write(consumerName).write("\",consumer_id=\"")
- .write(consumerId).write("\"} ");
- stream.write(value);
- appendEndings(stream);
+ private static void writeCompactionStat(Map<String, ByteBuf> allMetrics, String metricName,
+ Number value, Optional<CompactorMXBean> compactorMXBean,
+ String cluster, String namespace, String topic,
+ boolean splitTopicAndPartitionIndexLabel) {
+ boolean hasCompaction = compactorMXBean.flatMap(mxBean -> mxBean.getCompactionRecordForTopic(topic))
+ .isPresent();
+ if (hasCompaction) {
+ ByteBuf buffer = writeGaugeType(allMetrics, metricName);
+ writeTopicSample(buffer, metricName, value, cluster, namespace, topic, splitTopicAndPartitionIndexLabel);
+ }
}
- private static void metricWithRemoteCluster(SimpleTextOutputStream stream, String cluster, String namespace,
- String topic, String name, String remoteCluster, double value, boolean splitTopicAndPartitionIndexLabel) {
- metricType(stream, name);
- appendRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel)
- .write("\",remote_cluster=\"").write(remoteCluster).write("\"} ");
- stream.write(value);
- appendEndings(stream);
+ static void writeMetricWithBrokerDefault(Map<String, ByteBuf> allMetrics, String metricName, Number value,
+ String cluster, String namespace, String topic,
+ boolean splitTopicAndPartitionIndexLabel) {
+ ByteBuf buffer = writeGaugeTypeWithBrokerDefault(allMetrics, metricName, cluster);
+ writeTopicSample(buffer, metricName, value, cluster, namespace, topic, splitTopicAndPartitionIndexLabel);
}
- private static SimpleTextOutputStream appendRequiredLabels(SimpleTextOutputStream stream, String cluster,
- String namespace, String topic, String name, boolean splitTopicAndPartitionIndexLabel) {
- stream.write(name).write("{cluster=\"").write(cluster).write("\",namespace=\"").write(namespace);
+ static void writeTopicSample(ByteBuf buffer, String metricName, Number value, String cluster,
+ String namespace, String topic, boolean splitTopicAndPartitionIndexLabel,
+ String... extraLabelsAndValues) {
Review Comment:
@codelipenghui is ok with this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1210516801
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1172096693
@asafm That makes sense, I hadn't thought about the impact with that volume of topics (we currently have relatively few in comparison). I will look at using `ByteBuf` as suggested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] asafm commented on a diff in pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
asafm commented on code in PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#discussion_r913071081
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/NamespaceStatsAggregator.java:
##########
@@ -296,161 +306,264 @@ private static void getTopicStats(Topic topic, TopicStats stats, boolean include
});
}
- private static void printDefaultBrokerStats(SimpleTextOutputStream stream, String cluster) {
- // Print metrics with 0 values. This is necessary to have the available brokers being
- // reported in the brokers dashboard even if they don't have any topic or traffic
- metric(stream, cluster, "pulsar_topics_count", 0);
- metric(stream, cluster, "pulsar_subscriptions_count", 0);
- metric(stream, cluster, "pulsar_producers_count", 0);
- metric(stream, cluster, "pulsar_consumers_count", 0);
- metric(stream, cluster, "pulsar_rate_in", 0);
- metric(stream, cluster, "pulsar_rate_out", 0);
- metric(stream, cluster, "pulsar_throughput_in", 0);
- metric(stream, cluster, "pulsar_throughput_out", 0);
- metric(stream, cluster, "pulsar_storage_size", 0);
- metric(stream, cluster, "pulsar_storage_logical_size", 0);
- metric(stream, cluster, "pulsar_storage_write_rate", 0);
- metric(stream, cluster, "pulsar_storage_read_rate", 0);
- metric(stream, cluster, "pulsar_msg_backlog", 0);
+ private static void printTopicsCountStats(Map<String, ByteBuf> allMetrics, Map<String, Long> namespaceTopicsCount,
+ String cluster) {
+ namespaceTopicsCount.forEach((ns, topicCount) ->
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_topics_count", topicCount, cluster, ns)
+ );
}
- private static void printTopicsCountStats(SimpleTextOutputStream stream, String cluster, String namespace,
- LongAdder topicsCount) {
- metric(stream, cluster, namespace, "pulsar_topics_count", topicsCount.sum());
- }
+ private static void printNamespaceStats(Map<String, ByteBuf> allMetrics, AggregatedNamespaceStats stats,
+ String cluster, String namespace) {
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_topics_count", stats.topicsCount, cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_subscriptions_count", stats.subscriptionsCount, cluster,
+ namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_producers_count", stats.producersCount, cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_consumers_count", stats.consumersCount, cluster, namespace);
+
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_rate_in", stats.rateIn, cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_rate_out", stats.rateOut, cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_throughput_in", stats.throughputIn, cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_throughput_out", stats.throughputOut, cluster, namespace);
+ writeMetric(allMetrics, "pulsar_consumer_msg_ack_rate", stats.messageAckRate, cluster, namespace);
+
+ writeMetric(allMetrics, "pulsar_in_bytes_total", stats.bytesInCounter, cluster, namespace);
+ writeMetric(allMetrics, "pulsar_in_messages_total", stats.msgInCounter, cluster, namespace);
+ writeMetric(allMetrics, "pulsar_out_bytes_total", stats.bytesOutCounter, cluster, namespace);
+ writeMetric(allMetrics, "pulsar_out_messages_total", stats.msgOutCounter, cluster, namespace);
+
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_storage_size", stats.managedLedgerStats.storageSize, cluster,
+ namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_storage_logical_size",
+ stats.managedLedgerStats.storageLogicalSize, cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_backlog_size", stats.managedLedgerStats.backlogSize, cluster,
+ namespace);
+ writeMetric(allMetrics, "pulsar_storage_offloaded_size",
+ stats.managedLedgerStats.offloadedStorageUsed, cluster, namespace);
+
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_storage_write_rate", stats.managedLedgerStats.storageWriteRate,
+ cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_storage_read_rate", stats.managedLedgerStats.storageReadRate,
+ cluster, namespace);
+
+ writeMetric(allMetrics, "pulsar_subscription_delayed", stats.msgDelayed, cluster, namespace);
+
+ writePulsarMsgBacklog(allMetrics, stats.msgBacklog, cluster, namespace);
- private static void printNamespaceStats(SimpleTextOutputStream stream, String cluster, String namespace,
- AggregatedNamespaceStats stats) {
- metric(stream, cluster, namespace, "pulsar_topics_count", stats.topicsCount);
- metric(stream, cluster, namespace, "pulsar_subscriptions_count", stats.subscriptionsCount);
- metric(stream, cluster, namespace, "pulsar_producers_count", stats.producersCount);
- metric(stream, cluster, namespace, "pulsar_consumers_count", stats.consumersCount);
+ stats.managedLedgerStats.storageWriteLatencyBuckets.refresh();
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_0_5",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[0], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_1",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[1], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_5",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[2], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_10",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[3], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_20",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[4], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_50",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[5], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_100",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[6], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_200",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[7], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_1000",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[8], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_overflow",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[9], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_count",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getCount(), cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_sum",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getSum(), cluster, namespace);
- metric(stream, cluster, namespace, "pulsar_rate_in", stats.rateIn);
- metric(stream, cluster, namespace, "pulsar_rate_out", stats.rateOut);
- metric(stream, cluster, namespace, "pulsar_throughput_in", stats.throughputIn);
- metric(stream, cluster, namespace, "pulsar_throughput_out", stats.throughputOut);
- metric(stream, cluster, namespace, "pulsar_consumer_msg_ack_rate", stats.messageAckRate);
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.refresh();
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_0_5",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[0], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_1",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[1], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_5",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[2], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_10",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[3], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_20",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[4], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_50",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[5], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_100",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[6], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_200",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[7], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_1000",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[8], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_overflow",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[9], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_count",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getCount(), cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_sum",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getSum(), cluster, namespace);
- metric(stream, cluster, namespace, "pulsar_in_bytes_total", stats.bytesInCounter);
- metric(stream, cluster, namespace, "pulsar_in_messages_total", stats.msgInCounter);
- metric(stream, cluster, namespace, "pulsar_out_bytes_total", stats.bytesOutCounter);
- metric(stream, cluster, namespace, "pulsar_out_messages_total", stats.msgOutCounter);
+ stats.managedLedgerStats.entrySizeBuckets.refresh();
+ writeMetric(allMetrics, "pulsar_entry_size_le_128", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[0],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_512", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[1],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_1_kb", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[2],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_2_kb", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[3],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_4_kb", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[4],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_16_kb", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[5],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_100_kb",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[6], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_1_mb", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[7],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_overflow",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[8], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_count",
+ stats.managedLedgerStats.entrySizeBuckets.getCount(), cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_sum",
+ stats.managedLedgerStats.entrySizeBuckets.getSum(), cluster, namespace);
+
+ writeReplicationStat(allMetrics, "pulsar_replication_rate_in", stats,
+ replStats -> replStats.msgRateIn, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_rate_out", stats,
+ replStats -> replStats.msgRateOut, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_throughput_in", stats,
+ replStats -> replStats.msgThroughputIn, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_throughput_out", stats,
+ replStats -> replStats.msgThroughputOut, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_backlog", stats,
+ replStats -> replStats.replicationBacklog, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_connected_count", stats,
+ replStats -> replStats.connectedCount, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_rate_expired", stats,
+ replStats -> replStats.msgRateExpired, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_delay_in_seconds", stats,
+ replStats -> replStats.replicationDelayInSeconds, cluster, namespace);
+ }
- metric(stream, cluster, namespace, "pulsar_storage_size", stats.managedLedgerStats.storageSize);
- metric(stream, cluster, namespace, "pulsar_storage_logical_size", stats.managedLedgerStats.storageLogicalSize);
- metric(stream, cluster, namespace, "pulsar_storage_backlog_size", stats.managedLedgerStats.backlogSize);
- metric(stream, cluster, namespace, "pulsar_storage_offloaded_size",
- stats.managedLedgerStats.offloadedStorageUsed);
+ private static void writeMetricWithBrokerDefault(Map<String, ByteBuf> allMetrics, String metricName, Number value,
+ String cluster, String namespace) {
+ ByteBuf buffer = writeGaugeTypeWithBrokerDefault(allMetrics, metricName, cluster);
+ writeSample(buffer, metricName, value, "cluster", cluster, "namespace", namespace);
+ }
- metric(stream, cluster, namespace, "pulsar_storage_write_rate", stats.managedLedgerStats.storageWriteRate);
- metric(stream, cluster, namespace, "pulsar_storage_read_rate", stats.managedLedgerStats.storageReadRate);
+ private static void writePulsarMsgBacklog(Map<String, ByteBuf> allMetrics, Number value,
+ String cluster, String namespace) {
+ ByteBuf buffer = writeGaugeTypeWithBrokerDefault(allMetrics, "pulsar_msg_backlog", cluster);
+ writeSample(buffer, "pulsar_msg_backlog", value, "cluster", cluster, "namespace", namespace, "remote_cluster",
+ "local");
+ }
- metric(stream, cluster, namespace, "pulsar_subscription_delayed", stats.msgDelayed);
+ private static void writeMetric(Map<String, ByteBuf> allMetrics, String metricName, Number value, String cluster,
+ String namespace) {
+ ByteBuf buffer = writeGaugeType(allMetrics, metricName);
+ writeSample(buffer, metricName, value, "cluster", cluster, "namespace", namespace);
+ }
- metricWithRemoteCluster(stream, cluster, namespace, "pulsar_msg_backlog", "local", stats.msgBacklog);
+ private static void writeReplicationStat(Map<String, ByteBuf> allMetrics, String metricName,
+ AggregatedNamespaceStats namespaceStats,
+ Function<AggregatedReplicationStats, Number> replStatsFunction,
+ String cluster, String namespace) {
+ if (!namespaceStats.replicationStats.isEmpty()) {
+ ByteBuf buffer = writeGaugeType(allMetrics, metricName);
+ namespaceStats.replicationStats.forEach((remoteCluster, replStats) ->
+ writeSample(buffer, metricName, replStatsFunction.apply(replStats),
+ "cluster", cluster,
+ "namespace", namespace,
+ "remote_cluster", remoteCluster)
+ );
+ }
+ }
- stats.managedLedgerStats.storageWriteLatencyBuckets.refresh();
- long[] latencyBuckets = stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets();
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_0_5", latencyBuckets[0]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_1", latencyBuckets[1]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_5", latencyBuckets[2]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_10", latencyBuckets[3]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_20", latencyBuckets[4]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_50", latencyBuckets[5]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_100", latencyBuckets[6]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_200", latencyBuckets[7]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_1000", latencyBuckets[8]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_overflow", latencyBuckets[9]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_count",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getCount());
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_sum",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getSum());
+ static ByteBuf writeGaugeType(Map<String, ByteBuf> allMetrics, String metricName) {
+ if (!allMetrics.containsKey(metricName)) {
+ ByteBuf buffer = UnpooledByteBufAllocator.DEFAULT.compositeDirectBuffer(MAX_COMPONENTS);
Review Comment:
@merlimat , @tjiuming - you think we should use Direct Buffer here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1229201607
@Jason918 I don't believe the failures are to do with the changes in this PR, would be grateful for help confirming this or how I can get these to pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] asafm commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
asafm commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1231496642
@codelipenghui @eolivelli Let's merge this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
mattisonchao commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1244813725
Hello @marksilcox
It looks like we got many conflicts when cherry-picking it to branch-2.9.
Would you mind pushing a PR to branch-2.9? (To avoid cherry-picking involving bugs)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1221507027
> @marksilcox Your work was great. We can settle for opening an issue and linking this PR to it - what do you say?
@asafm there is already an open issue (#8407) for this, or is a new one required?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jason918 merged pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
Jason918 merged PR #15558:
URL: https://github.com/apache/pulsar/pull/15558
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on a diff in pull request #15558: [fix][broker/functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
eolivelli commented on code in PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#discussion_r872336085
##########
conf/standalone.conf:
##########
@@ -1099,4 +1099,4 @@ configurationStoreServers=
# than this limit then broker will persist unacked ranges into bookkeeper to avoid additional data overhead into
# zookeeper.
# Deprecated: use managedLedgerMaxUnackedRangesToPersistInMetadataStore
-managedLedgerMaxUnackedRangesToPersistInZooKeeper=-1
\ No newline at end of file
+managedLedgerMaxUnackedRangesToPersistInZooKeeper=-1
Review Comment:
nit: please revert unnecessary changes
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/TopicStats.java:
##########
@@ -106,269 +112,270 @@ static void resetTypes() {
metricWithTypeDefinition.clear();
}
- static void printTopicStats(SimpleTextOutputStream stream, String cluster, String namespace, String topic,
+ static void printTopicStats(Map<String, List<String>> metrics, String cluster, String namespace, String topic,
Review Comment:
it looks like this change is not only about the Function Worker.
Please update the title
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/TopicStats.java:
##########
@@ -382,77 +389,102 @@ static void metricType(SimpleTextOutputStream stream, String name) {
}
- private static void metric(SimpleTextOutputStream stream, String cluster, String namespace, String topic,
- String name, double value, boolean splitTopicAndPartitionIndexLabel) {
- metricType(stream, name);
- appendRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel).write("\"} ");
- stream.write(value);
- appendEndings(stream);
+ private static void metric(Map<String, List<String>> metrics, String cluster, String namespace, String topic,
+ String name, double value, boolean splitTopicAndPartitionIndexLabel) {
+ writeMetric(metrics, name, stream -> {
+ writeRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel);
+ stream.write("\"} ")
+ .write(value);
+ writeEndings(stream);
+ });
}
- private static void metric(SimpleTextOutputStream stream, String cluster, String namespace, String topic,
- String subscription, String name, long value, boolean splitTopicAndPartitionIndexLabel) {
- metricType(stream, name);
- appendRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel)
- .write("\",subscription=\"").write(subscription).write("\"} ");
- stream.write(value);
- appendEndings(stream);
+ private static void metric(Map<String, List<String>> metrics, String cluster, String namespace, String topic,
+ String subscription, String name, long value, boolean splitTopicAndPartitionIndexLabel) {
+ writeMetric(metrics, name, stream -> {
+ writeRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel);
+ stream.write("\",subscription=\"").write(subscription)
+ .write("\"} ")
+ .write(value);
+ writeEndings(stream);
+ });
+ }
+
+ private static void metric(Map<String, List<String>> metrics, String cluster, String namespace, String topic,
+ String producerName, long produceId, String name, double value,
+ boolean splitTopicAndPartitionIndexLabel) {
+ writeMetric(metrics, name, stream -> {
+ writeRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel);
+ stream.write("\",producer_name=\"").write(producerName)
+ .write("\",producer_id=\"").write(produceId)
+ .write("\"} ")
+ .write(value);
+ writeEndings(stream);
+ });
}
- private static void metric(SimpleTextOutputStream stream, String cluster, String namespace, String topic,
- String producerName, long produceId, String name, double value, boolean splitTopicAndPartitionIndexLabel) {
- metricType(stream, name);
- appendRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel)
- .write("\",producer_name=\"").write(producerName)
- .write("\",producer_id=\"").write(produceId).write("\"} ");
- stream.write(value);
- appendEndings(stream);
+ private static void metric(Map<String, List<String>> metrics, String cluster, String namespace, String topic,
+ String subscription, String name, double value,
+ boolean splitTopicAndPartitionIndexLabel) {
+ writeMetric(metrics, name, stream -> {
+ writeRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel);
+ stream.write("\",subscription=\"").write(subscription)
+ .write("\"} ")
+ .write(value);
+ writeEndings(stream);
+ });
}
- private static void metric(SimpleTextOutputStream stream, String cluster, String namespace, String topic,
- String subscription, String name, double value, boolean splitTopicAndPartitionIndexLabel) {
- metricType(stream, name);
- appendRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel)
- .write("\",subscription=\"").write(subscription).write("\"} ");
- stream.write(value);
- appendEndings(stream);
+ private static void metric(Map<String, List<String>> metrics, String cluster, String namespace, String topic,
+ String subscription, String consumerName, long consumerId, String name, long value,
+ boolean splitTopicAndPartitionIndexLabel) {
+ writeMetric(metrics, name, stream -> {
+ writeRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel);
+ stream.write("\",subscription=\"").write(subscription)
+ .write("\",consumer_name=\"").write(consumerName).write("\",consumer_id=\"").write(consumerId)
+ .write("\"} ")
+ .write(value);
+ writeEndings(stream);
+ });
}
- private static void metric(SimpleTextOutputStream stream, String cluster, String namespace, String topic,
- String subscription, String consumerName, long consumerId, String name, long value,
- boolean splitTopicAndPartitionIndexLabel) {
- metricType(stream, name);
- appendRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel)
- .write("\",subscription=\"").write(subscription)
- .write("\",consumer_name=\"").write(consumerName).write("\",consumer_id=\"").write(consumerId)
- .write("\"} ");
- stream.write(value);
- appendEndings(stream);
+ private static void metric(Map<String, List<String>> metrics, String cluster, String namespace, String topic,
+ String subscription, String consumerName, long consumerId, String name, double value,
+ boolean splitTopicAndPartitionIndexLabel) {
+ writeMetric(metrics, name, stream -> {
+ writeRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel);
+ stream.write("\",subscription=\"").write(subscription)
+ .write("\",consumer_name=\"").write(consumerName).write("\",consumer_id=\"")
+ .write(consumerId).write("\"} ")
+ .write(value);
+ writeEndings(stream);
+ });
}
- private static void metric(SimpleTextOutputStream stream, String cluster, String namespace, String topic,
- String subscription, String consumerName, long consumerId, String name, double value,
- boolean splitTopicAndPartitionIndexLabel) {
- metricType(stream, name);
- appendRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel)
- .write("\",subscription=\"").write(subscription)
- .write("\",consumer_name=\"").write(consumerName).write("\",consumer_id=\"")
- .write(consumerId).write("\"} ");
- stream.write(value);
- appendEndings(stream);
+ private static void metricWithRemoteCluster(Map<String, List<String>> metrics, String cluster, String namespace,
+ String topic, String name, String remoteCluster, double value,
+ boolean splitTopicAndPartitionIndexLabel) {
+ writeMetric(metrics, name, stream -> {
+ writeRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel);
+ stream.write("\",remote_cluster=\"").write(remoteCluster).write("\"} ").write(value);
+ writeEndings(stream);
+ });
}
- private static void metricWithRemoteCluster(SimpleTextOutputStream stream, String cluster, String namespace,
- String topic, String name, String remoteCluster, double value, boolean splitTopicAndPartitionIndexLabel) {
- metricType(stream, name);
- appendRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel)
- .write("\",remote_cluster=\"").write(remoteCluster).write("\"} ");
- stream.write(value);
- appendEndings(stream);
+ private static void writeMetric(Map<String, List<String>> metrics, String name,
+ Consumer<SimpleTextOutputStream> metricWriter) {
+ ByteBuf buf = ByteBufAllocator.DEFAULT.heapBuffer();
Review Comment:
it looks like we are doing more memory allocations (even if you call 'buf.release()')
is it possible to move back to creating the SimpleTextOutputStream only once ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1128946935
@eolivelli could you please approve the workflows and review again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] asafm commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
asafm commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1163261876
> The problem we have is that we need to collect by metric name, and not by topic/namespace as is currently happening. So for a metric such as `pulsar_rate_in` whether `includeTopics` is enabled or not we need all values of this metric grouped under a single type header.
>
> This means whatever is used needs to store each metric in a way so other namespaces or topics can add there values. I guess what I need to find is the most memory efficient way of managing this.
I believe you can stick to the same data structures used now.
Just when printing it, you iterate over the different data structures.
For example, pulsa_rate_in, you iterate over the the TopicStats, grab the value from there and print it.
Do that for every metric.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1165268082
/pulsarbot run unit-tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] asafm commented on a diff in pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
asafm commented on code in PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#discussion_r915016056
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricStreams.java:
##########
@@ -0,0 +1,74 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.stats.prometheus;
+
+import io.netty.buffer.ByteBufAllocator;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pulsar.common.util.SimpleTextOutputStream;
+
+/**
+ * Helper class to ensure that metrics of the same name are grouped together under the same TYPE header when written.
+ */
+public class PrometheusMetricStreams {
+ private final Map<String, SimpleTextOutputStream> metricStreamMap = new HashMap<>();
+
+ /**
+ * Write the given metric and sample value to the stream. Will write #TYPE header if metric not seen before.
+ * @param metricName name of the metric.
+ * @param value value of the sample
+ * @param labelsAndValuesArray varargs of label and label value
+ */
+ void writeSample(String metricName, Number value, String... labelsAndValuesArray) {
+ SimpleTextOutputStream stream = initGaugeType(metricName);
+ stream.write(metricName).write('{');
+ for (int i = 0; i < labelsAndValuesArray.length; i += 2) {
+ stream.write(labelsAndValuesArray[i]).write("=\"").write(labelsAndValuesArray[i + 1]).write('\"');
+ if (i + 2 != labelsAndValuesArray.length) {
+ stream.write(',');
+ }
+ }
+ stream.write("\"} ").write(value).write(' ').write(System.currentTimeMillis()).write('\n');
Review Comment:
@marksilcox I missed that appendEnding somehow. Thanks
@hangc0276 @merlimat - I think we can drop time since it's optional and it will save bandwidth - WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1157916810
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1136881680
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] asafm commented on a diff in pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
asafm commented on code in PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#discussion_r938022919
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricStreams.java:
##########
@@ -0,0 +1,75 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.stats.prometheus;
+
+import io.netty.buffer.ByteBufAllocator;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pulsar.common.util.SimpleTextOutputStream;
+
+/**
+ * Helper class to ensure that metrics of the same name are grouped together under the same TYPE header when written.
+ * Those are the requirements of the
+ * <a href="https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md#grouping-and-sorting">Prometheus Exposition Format</a>.
+ */
+public class PrometheusMetricStreams {
+ private final Map<String, SimpleTextOutputStream> metricStreamMap = new HashMap<>();
+
+ /**
+ * Write the given metric and sample value to the stream. Will write #TYPE header if metric not seen before.
+ * @param metricName name of the metric.
+ * @param value value of the sample
+ * @param labelsAndValuesArray varargs of label and label value
+ */
+ void writeSample(String metricName, Number value, String... labelsAndValuesArray) {
+ SimpleTextOutputStream stream = initGaugeType(metricName);
+ stream.write(metricName).write('{');
+ for (int i = 0; i < labelsAndValuesArray.length; i += 2) {
+ stream.write(labelsAndValuesArray[i]).write("=\"").write(labelsAndValuesArray[i + 1]).write('\"');
+ if (i + 2 != labelsAndValuesArray.length) {
+ stream.write(',');
+ }
+ }
+ stream.write("} ").write(value).write(' ').write(System.currentTimeMillis()).write('\n');
+ }
+
+ /**
+ * Flush all the stored metrics to the supplied stream.
+ * @param stream the stream to write to.
+ */
+ void flushAllToStream(SimpleTextOutputStream stream) {
+ metricStreamMap.values().forEach(s -> stream.write(s.getBuffer()));
+ }
+
+ /**
+ * Release all the streams to clean up resources.
+ */
+ void releaseAll() {
+ metricStreamMap.values().forEach(s -> s.getBuffer().release());
+ metricStreamMap.clear();
+ }
+
+ private SimpleTextOutputStream initGaugeType(String metricName) {
+ return metricStreamMap.computeIfAbsent(metricName, s -> {
+ SimpleTextOutputStream stream = new SimpleTextOutputStream(ByteBufAllocator.DEFAULT.directBuffer());
Review Comment:
It seems that needs fixing, and use `PulsarByteBufAllocator`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] asafm commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
asafm commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1207906842
All looks good on my end :)
I'll try to find someone with write permission to wrap this up
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jason918 commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
Jason918 commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1229150078
@marksilcox Can you solve the CI failure?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on pull request #15558: [fix][broker] Ensure prometheus metrics are grouped by type (#8407)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1124681663
@codelipenghui can you look at this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] hangc0276 commented on a diff in pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on code in PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#discussion_r936293586
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricStreams.java:
##########
@@ -0,0 +1,74 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.stats.prometheus;
+
+import io.netty.buffer.ByteBufAllocator;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pulsar.common.util.SimpleTextOutputStream;
+
+/**
+ * Helper class to ensure that metrics of the same name are grouped together under the same TYPE header when written.
+ */
+public class PrometheusMetricStreams {
+ private final Map<String, SimpleTextOutputStream> metricStreamMap = new HashMap<>();
+
+ /**
+ * Write the given metric and sample value to the stream. Will write #TYPE header if metric not seen before.
+ * @param metricName name of the metric.
+ * @param value value of the sample
+ * @param labelsAndValuesArray varargs of label and label value
+ */
+ void writeSample(String metricName, Number value, String... labelsAndValuesArray) {
+ SimpleTextOutputStream stream = initGaugeType(metricName);
+ stream.write(metricName).write('{');
+ for (int i = 0; i < labelsAndValuesArray.length; i += 2) {
+ stream.write(labelsAndValuesArray[i]).write("=\"").write(labelsAndValuesArray[i + 1]).write('\"');
+ if (i + 2 != labelsAndValuesArray.length) {
+ stream.write(',');
+ }
+ }
+ stream.write("\"} ").write(value).write(' ').write(System.currentTimeMillis()).write('\n');
Review Comment:
It's ok for me to drop the timestamp item. When the Prometheus server pulls the metrics, it will add the received timestamp, that's enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] asafm commented on a diff in pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
asafm commented on code in PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#discussion_r935251401
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricStreams.java:
##########
@@ -0,0 +1,75 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.stats.prometheus;
+
+import io.netty.buffer.ByteBufAllocator;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pulsar.common.util.SimpleTextOutputStream;
+
+/**
+ * Helper class to ensure that metrics of the same name are grouped together under the same TYPE header when written.
+ * Those are the requirements of the
+ * <a href="https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md#grouping-and-sorting">Prometheus Exposition Format</a>.
+ */
+public class PrometheusMetricStreams {
+ private final Map<String, SimpleTextOutputStream> metricStreamMap = new HashMap<>();
+
+ /**
+ * Write the given metric and sample value to the stream. Will write #TYPE header if metric not seen before.
+ * @param metricName name of the metric.
+ * @param value value of the sample
+ * @param labelsAndValuesArray varargs of label and label value
+ */
+ void writeSample(String metricName, Number value, String... labelsAndValuesArray) {
+ SimpleTextOutputStream stream = initGaugeType(metricName);
+ stream.write(metricName).write('{');
+ for (int i = 0; i < labelsAndValuesArray.length; i += 2) {
+ stream.write(labelsAndValuesArray[i]).write("=\"").write(labelsAndValuesArray[i + 1]).write('\"');
+ if (i + 2 != labelsAndValuesArray.length) {
+ stream.write(',');
+ }
+ }
+ stream.write("\"} ").write(value).write(' ').write(System.currentTimeMillis()).write('\n');
Review Comment:
Just double checking here - the `stream.write("\"{ ")` - what's `"` closing here?
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/NamespaceStatsAggregator.java:
##########
@@ -296,161 +301,192 @@ private static void getTopicStats(Topic topic, TopicStats stats, boolean include
});
}
- private static void printDefaultBrokerStats(SimpleTextOutputStream stream, String cluster) {
+ private static void printDefaultBrokerStats(PrometheusMetricStreams stream, String cluster) {
// Print metrics with 0 values. This is necessary to have the available brokers being
// reported in the brokers dashboard even if they don't have any topic or traffic
- metric(stream, cluster, "pulsar_topics_count", 0);
- metric(stream, cluster, "pulsar_subscriptions_count", 0);
- metric(stream, cluster, "pulsar_producers_count", 0);
- metric(stream, cluster, "pulsar_consumers_count", 0);
- metric(stream, cluster, "pulsar_rate_in", 0);
- metric(stream, cluster, "pulsar_rate_out", 0);
- metric(stream, cluster, "pulsar_throughput_in", 0);
- metric(stream, cluster, "pulsar_throughput_out", 0);
- metric(stream, cluster, "pulsar_storage_size", 0);
- metric(stream, cluster, "pulsar_storage_logical_size", 0);
- metric(stream, cluster, "pulsar_storage_write_rate", 0);
- metric(stream, cluster, "pulsar_storage_read_rate", 0);
- metric(stream, cluster, "pulsar_msg_backlog", 0);
+ writeMetric(stream, "pulsar_topics_count", 0, cluster);
+ writeMetric(stream, "pulsar_subscriptions_count", 0, cluster);
+ writeMetric(stream, "pulsar_producers_count", 0, cluster);
+ writeMetric(stream, "pulsar_consumers_count", 0, cluster);
+ writeMetric(stream, "pulsar_rate_in", 0, cluster);
+ writeMetric(stream, "pulsar_rate_out", 0, cluster);
+ writeMetric(stream, "pulsar_throughput_in", 0, cluster);
+ writeMetric(stream, "pulsar_throughput_out", 0, cluster);
+ writeMetric(stream, "pulsar_storage_size", 0, cluster);
+ writeMetric(stream, "pulsar_storage_logical_size", 0, cluster);
+ writeMetric(stream, "pulsar_storage_write_rate", 0, cluster);
+ writeMetric(stream, "pulsar_storage_read_rate", 0, cluster);
+ writeMetric(stream, "pulsar_msg_backlog", 0, cluster);
}
- private static void printTopicsCountStats(SimpleTextOutputStream stream, String cluster, String namespace,
- LongAdder topicsCount) {
- metric(stream, cluster, namespace, "pulsar_topics_count", topicsCount.sum());
+ private static void printTopicsCountStats(PrometheusMetricStreams stream, Map<String, Long> namespaceTopicsCount,
+ String cluster) {
+ namespaceTopicsCount.forEach(
+ (ns, topicCount) -> writeMetric(stream, "pulsar_topics_count", topicCount, cluster, ns)
+ );
}
- private static void printNamespaceStats(SimpleTextOutputStream stream, String cluster, String namespace,
- AggregatedNamespaceStats stats) {
- metric(stream, cluster, namespace, "pulsar_topics_count", stats.topicsCount);
- metric(stream, cluster, namespace, "pulsar_subscriptions_count", stats.subscriptionsCount);
- metric(stream, cluster, namespace, "pulsar_producers_count", stats.producersCount);
- metric(stream, cluster, namespace, "pulsar_consumers_count", stats.consumersCount);
-
- metric(stream, cluster, namespace, "pulsar_rate_in", stats.rateIn);
- metric(stream, cluster, namespace, "pulsar_rate_out", stats.rateOut);
- metric(stream, cluster, namespace, "pulsar_throughput_in", stats.throughputIn);
- metric(stream, cluster, namespace, "pulsar_throughput_out", stats.throughputOut);
- metric(stream, cluster, namespace, "pulsar_consumer_msg_ack_rate", stats.messageAckRate);
-
- metric(stream, cluster, namespace, "pulsar_in_bytes_total", stats.bytesInCounter);
- metric(stream, cluster, namespace, "pulsar_in_messages_total", stats.msgInCounter);
- metric(stream, cluster, namespace, "pulsar_out_bytes_total", stats.bytesOutCounter);
- metric(stream, cluster, namespace, "pulsar_out_messages_total", stats.msgOutCounter);
-
- metric(stream, cluster, namespace, "pulsar_storage_size", stats.managedLedgerStats.storageSize);
- metric(stream, cluster, namespace, "pulsar_storage_logical_size", stats.managedLedgerStats.storageLogicalSize);
- metric(stream, cluster, namespace, "pulsar_storage_backlog_size", stats.managedLedgerStats.backlogSize);
- metric(stream, cluster, namespace, "pulsar_storage_offloaded_size",
- stats.managedLedgerStats.offloadedStorageUsed);
-
- metric(stream, cluster, namespace, "pulsar_storage_write_rate", stats.managedLedgerStats.storageWriteRate);
- metric(stream, cluster, namespace, "pulsar_storage_read_rate", stats.managedLedgerStats.storageReadRate);
-
- metric(stream, cluster, namespace, "pulsar_subscription_delayed", stats.msgDelayed);
-
- metricWithRemoteCluster(stream, cluster, namespace, "pulsar_msg_backlog", "local", stats.msgBacklog);
+ private static void printNamespaceStats(PrometheusMetricStreams stream, AggregatedNamespaceStats stats,
+ String cluster, String namespace) {
+ writeMetric(stream, "pulsar_topics_count", stats.topicsCount, cluster, namespace);
+ writeMetric(stream, "pulsar_subscriptions_count", stats.subscriptionsCount, cluster,
+ namespace);
+ writeMetric(stream, "pulsar_producers_count", stats.producersCount, cluster, namespace);
+ writeMetric(stream, "pulsar_consumers_count", stats.consumersCount, cluster, namespace);
+
+ writeMetric(stream, "pulsar_rate_in", stats.rateIn, cluster, namespace);
+ writeMetric(stream, "pulsar_rate_out", stats.rateOut, cluster, namespace);
+ writeMetric(stream, "pulsar_throughput_in", stats.throughputIn, cluster, namespace);
+ writeMetric(stream, "pulsar_throughput_out", stats.throughputOut, cluster, namespace);
+ writeMetric(stream, "pulsar_consumer_msg_ack_rate", stats.messageAckRate, cluster, namespace);
+
+ writeMetric(stream, "pulsar_in_bytes_total", stats.bytesInCounter, cluster, namespace);
+ writeMetric(stream, "pulsar_in_messages_total", stats.msgInCounter, cluster, namespace);
+ writeMetric(stream, "pulsar_out_bytes_total", stats.bytesOutCounter, cluster, namespace);
+ writeMetric(stream, "pulsar_out_messages_total", stats.msgOutCounter, cluster, namespace);
+
+ writeMetric(stream, "pulsar_storage_size", stats.managedLedgerStats.storageSize, cluster,
+ namespace);
+ writeMetric(stream, "pulsar_storage_logical_size",
+ stats.managedLedgerStats.storageLogicalSize, cluster, namespace);
+ writeMetric(stream, "pulsar_storage_backlog_size", stats.managedLedgerStats.backlogSize, cluster,
+ namespace);
+ writeMetric(stream, "pulsar_storage_offloaded_size",
+ stats.managedLedgerStats.offloadedStorageUsed, cluster, namespace);
+
+ writeMetric(stream, "pulsar_storage_write_rate", stats.managedLedgerStats.storageWriteRate,
+ cluster, namespace);
+ writeMetric(stream, "pulsar_storage_read_rate", stats.managedLedgerStats.storageReadRate,
+ cluster, namespace);
+
+ writeMetric(stream, "pulsar_subscription_delayed", stats.msgDelayed, cluster, namespace);
+
+ writePulsarMsgBacklog(stream, stats.msgBacklog, cluster, namespace);
stats.managedLedgerStats.storageWriteLatencyBuckets.refresh();
- long[] latencyBuckets = stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets();
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_0_5", latencyBuckets[0]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_1", latencyBuckets[1]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_5", latencyBuckets[2]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_10", latencyBuckets[3]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_20", latencyBuckets[4]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_50", latencyBuckets[5]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_100", latencyBuckets[6]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_200", latencyBuckets[7]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_1000", latencyBuckets[8]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_overflow", latencyBuckets[9]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_count",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getCount());
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_sum",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getSum());
+ writeMetric(stream, "pulsar_storage_write_latency_le_0_5",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[0], cluster, namespace);
Review Comment:
Why not call `getBuckets()` once and re-use it below like it was? I think it's much easier to read that way. Relevant to all get buckets below
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricStreams.java:
##########
@@ -0,0 +1,74 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.stats.prometheus;
+
+import io.netty.buffer.ByteBufAllocator;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pulsar.common.util.SimpleTextOutputStream;
+
+/**
+ * Helper class to ensure that metrics of the same name are grouped together under the same TYPE header when written.
+ */
+public class PrometheusMetricStreams {
+ private final Map<String, SimpleTextOutputStream> metricStreamMap = new HashMap<>();
+
+ /**
+ * Write the given metric and sample value to the stream. Will write #TYPE header if metric not seen before.
+ * @param metricName name of the metric.
+ * @param value value of the sample
+ * @param labelsAndValuesArray varargs of label and label value
+ */
+ void writeSample(String metricName, Number value, String... labelsAndValuesArray) {
+ SimpleTextOutputStream stream = initGaugeType(metricName);
+ stream.write(metricName).write('{');
+ for (int i = 0; i < labelsAndValuesArray.length; i += 2) {
+ stream.write(labelsAndValuesArray[i]).write("=\"").write(labelsAndValuesArray[i + 1]).write('\"');
+ if (i + 2 != labelsAndValuesArray.length) {
+ stream.write(',');
+ }
+ }
+ stream.write("\"} ").write(value).write(' ').write(System.currentTimeMillis()).write('\n');
Review Comment:
@tjiuming WDYT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] asafm commented on a diff in pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
asafm commented on code in PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#discussion_r912467065
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/metrics/PrometheusTextFormatUtil.java:
##########
@@ -141,29 +140,32 @@ private static void writeSum(Writer w, DataSketchesOpStatsLogger opStat, String
.append(Double.toString(opStat.getSum(success))).append('\n');
}
- public static void writeMetricsCollectedByPrometheusClient(Writer w, CollectorRegistry registry)
- throws IOException {
- Enumeration<MetricFamilySamples> metricFamilySamples = registry.metricFamilySamples();
- while (metricFamilySamples.hasMoreElements()) {
- MetricFamilySamples metricFamily = metricFamilySamples.nextElement();
-
- for (int i = 0; i < metricFamily.samples.size(); i++) {
- Sample sample = metricFamily.samples.get(i);
- w.write(sample.name);
- w.write('{');
- for (int j = 0; j < sample.labelNames.size(); j++) {
- if (j != 0) {
- w.write(", ");
+ public static void writeMetrics(SimpleTextOutputStream stream, Collection<MetricFamilySamples> familySamples) {
Review Comment:
`familySamples` --> `familySamplesCollection`
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/metrics/PrometheusTextFormatUtil.java:
##########
@@ -141,29 +140,32 @@ private static void writeSum(Writer w, DataSketchesOpStatsLogger opStat, String
.append(Double.toString(opStat.getSum(success))).append('\n');
}
- public static void writeMetricsCollectedByPrometheusClient(Writer w, CollectorRegistry registry)
- throws IOException {
- Enumeration<MetricFamilySamples> metricFamilySamples = registry.metricFamilySamples();
- while (metricFamilySamples.hasMoreElements()) {
- MetricFamilySamples metricFamily = metricFamilySamples.nextElement();
-
- for (int i = 0; i < metricFamily.samples.size(); i++) {
- Sample sample = metricFamily.samples.get(i);
- w.write(sample.name);
- w.write('{');
- for (int j = 0; j < sample.labelNames.size(); j++) {
- if (j != 0) {
- w.write(", ");
+ public static void writeMetrics(SimpleTextOutputStream stream, Collection<MetricFamilySamples> familySamples) {
+ for (MetricFamilySamples familySample : familySamples) {
Review Comment:
`familySample` --> `familySamples`
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/metrics/PrometheusTextFormatUtil.java:
##########
@@ -141,29 +140,32 @@ private static void writeSum(Writer w, DataSketchesOpStatsLogger opStat, String
.append(Double.toString(opStat.getSum(success))).append('\n');
}
- public static void writeMetricsCollectedByPrometheusClient(Writer w, CollectorRegistry registry)
- throws IOException {
- Enumeration<MetricFamilySamples> metricFamilySamples = registry.metricFamilySamples();
- while (metricFamilySamples.hasMoreElements()) {
- MetricFamilySamples metricFamily = metricFamilySamples.nextElement();
-
- for (int i = 0; i < metricFamily.samples.size(); i++) {
- Sample sample = metricFamily.samples.get(i);
- w.write(sample.name);
- w.write('{');
- for (int j = 0; j < sample.labelNames.size(); j++) {
- if (j != 0) {
- w.write(", ");
+ public static void writeMetrics(SimpleTextOutputStream stream, Collection<MetricFamilySamples> familySamples) {
+ for (MetricFamilySamples familySample : familySamples) {
+ stream.write("# TYPE ");
Review Comment:
Before we write this line, let's verify that `familySample` actually has samples inside it.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/NamespaceStatsAggregator.java:
##########
@@ -44,62 +51,65 @@
@Slf4j
public class NamespaceStatsAggregator {
- private static FastThreadLocal<AggregatedNamespaceStats> localNamespaceStats =
- new FastThreadLocal<AggregatedNamespaceStats>() {
+ private static final FastThreadLocal<AggregatedNamespaceStats> localNamespaceStats =
+ new FastThreadLocal<>() {
@Override
- protected AggregatedNamespaceStats initialValue() throws Exception {
+ protected AggregatedNamespaceStats initialValue() {
return new AggregatedNamespaceStats();
}
};
- private static FastThreadLocal<TopicStats> localTopicStats = new FastThreadLocal<TopicStats>() {
+ private static final FastThreadLocal<TopicStats> localTopicStats = new FastThreadLocal<>() {
@Override
- protected TopicStats initialValue() throws Exception {
+ protected TopicStats initialValue() {
return new TopicStats();
}
};
public static void generate(PulsarService pulsar, boolean includeTopicMetrics, boolean includeConsumerMetrics,
- boolean includeProducerMetrics, boolean splitTopicAndPartitionIndexLabel, SimpleTextOutputStream stream) {
+ boolean includeProducerMetrics, boolean splitTopicAndPartitionIndexLabel,
+ SimpleTextOutputStream stream) {
String cluster = pulsar.getConfiguration().getClusterName();
AggregatedNamespaceStats namespaceStats = localNamespaceStats.get();
- TopicStats.resetTypes();
TopicStats topicStats = localTopicStats.get();
-
- printDefaultBrokerStats(stream, cluster);
-
Optional<CompactorMXBean> compactorMXBean = getCompactorMXBean(pulsar);
LongAdder topicsCount = new LongAdder();
+ Map<String, ByteBuf> allMetrics = new HashMap<>();
Review Comment:
Namespace and topics stats is not the only place that prints metrics in topics granularity. We also have `TransactionAggregator` that has topic and subscription level metrics --> same as `AggregatedNamespaceStats` which means we need to do the same trick there as well.
Due to the above, and for better code design:
```
MetricStreams {
Map<String, SimpleTextOutputStream> metricNameToStreamMap
interface MetricWriter {
void write(SimpleTextOutputStream out)
}
write(metricName, metricWriter) {
metricStream = BytemetricNameToByteBufMap.computeIfAbsent(metricName, initStream)
metricWriter.write(metricStream)
}
flushAllToSingleStream()
}
```
Using SimpleTextOutputStream to avoid duplicating code from it as was done in two classes in this PR
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/NamespaceStatsAggregator.java:
##########
@@ -296,161 +306,264 @@ private static void getTopicStats(Topic topic, TopicStats stats, boolean include
});
}
- private static void printDefaultBrokerStats(SimpleTextOutputStream stream, String cluster) {
- // Print metrics with 0 values. This is necessary to have the available brokers being
- // reported in the brokers dashboard even if they don't have any topic or traffic
- metric(stream, cluster, "pulsar_topics_count", 0);
- metric(stream, cluster, "pulsar_subscriptions_count", 0);
- metric(stream, cluster, "pulsar_producers_count", 0);
- metric(stream, cluster, "pulsar_consumers_count", 0);
- metric(stream, cluster, "pulsar_rate_in", 0);
- metric(stream, cluster, "pulsar_rate_out", 0);
- metric(stream, cluster, "pulsar_throughput_in", 0);
- metric(stream, cluster, "pulsar_throughput_out", 0);
- metric(stream, cluster, "pulsar_storage_size", 0);
- metric(stream, cluster, "pulsar_storage_logical_size", 0);
- metric(stream, cluster, "pulsar_storage_write_rate", 0);
- metric(stream, cluster, "pulsar_storage_read_rate", 0);
- metric(stream, cluster, "pulsar_msg_backlog", 0);
+ private static void printTopicsCountStats(Map<String, ByteBuf> allMetrics, Map<String, Long> namespaceTopicsCount,
+ String cluster) {
+ namespaceTopicsCount.forEach((ns, topicCount) ->
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_topics_count", topicCount, cluster, ns)
+ );
}
- private static void printTopicsCountStats(SimpleTextOutputStream stream, String cluster, String namespace,
- LongAdder topicsCount) {
- metric(stream, cluster, namespace, "pulsar_topics_count", topicsCount.sum());
- }
+ private static void printNamespaceStats(Map<String, ByteBuf> allMetrics, AggregatedNamespaceStats stats,
+ String cluster, String namespace) {
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_topics_count", stats.topicsCount, cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_subscriptions_count", stats.subscriptionsCount, cluster,
+ namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_producers_count", stats.producersCount, cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_consumers_count", stats.consumersCount, cluster, namespace);
+
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_rate_in", stats.rateIn, cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_rate_out", stats.rateOut, cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_throughput_in", stats.throughputIn, cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_throughput_out", stats.throughputOut, cluster, namespace);
+ writeMetric(allMetrics, "pulsar_consumer_msg_ack_rate", stats.messageAckRate, cluster, namespace);
+
+ writeMetric(allMetrics, "pulsar_in_bytes_total", stats.bytesInCounter, cluster, namespace);
+ writeMetric(allMetrics, "pulsar_in_messages_total", stats.msgInCounter, cluster, namespace);
+ writeMetric(allMetrics, "pulsar_out_bytes_total", stats.bytesOutCounter, cluster, namespace);
+ writeMetric(allMetrics, "pulsar_out_messages_total", stats.msgOutCounter, cluster, namespace);
+
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_storage_size", stats.managedLedgerStats.storageSize, cluster,
+ namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_storage_logical_size",
+ stats.managedLedgerStats.storageLogicalSize, cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_backlog_size", stats.managedLedgerStats.backlogSize, cluster,
+ namespace);
+ writeMetric(allMetrics, "pulsar_storage_offloaded_size",
+ stats.managedLedgerStats.offloadedStorageUsed, cluster, namespace);
+
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_storage_write_rate", stats.managedLedgerStats.storageWriteRate,
+ cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_storage_read_rate", stats.managedLedgerStats.storageReadRate,
+ cluster, namespace);
+
+ writeMetric(allMetrics, "pulsar_subscription_delayed", stats.msgDelayed, cluster, namespace);
+
+ writePulsarMsgBacklog(allMetrics, stats.msgBacklog, cluster, namespace);
- private static void printNamespaceStats(SimpleTextOutputStream stream, String cluster, String namespace,
- AggregatedNamespaceStats stats) {
- metric(stream, cluster, namespace, "pulsar_topics_count", stats.topicsCount);
- metric(stream, cluster, namespace, "pulsar_subscriptions_count", stats.subscriptionsCount);
- metric(stream, cluster, namespace, "pulsar_producers_count", stats.producersCount);
- metric(stream, cluster, namespace, "pulsar_consumers_count", stats.consumersCount);
+ stats.managedLedgerStats.storageWriteLatencyBuckets.refresh();
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_0_5",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[0], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_1",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[1], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_5",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[2], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_10",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[3], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_20",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[4], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_50",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[5], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_100",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[6], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_200",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[7], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_1000",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[8], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_overflow",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[9], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_count",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getCount(), cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_sum",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getSum(), cluster, namespace);
- metric(stream, cluster, namespace, "pulsar_rate_in", stats.rateIn);
- metric(stream, cluster, namespace, "pulsar_rate_out", stats.rateOut);
- metric(stream, cluster, namespace, "pulsar_throughput_in", stats.throughputIn);
- metric(stream, cluster, namespace, "pulsar_throughput_out", stats.throughputOut);
- metric(stream, cluster, namespace, "pulsar_consumer_msg_ack_rate", stats.messageAckRate);
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.refresh();
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_0_5",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[0], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_1",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[1], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_5",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[2], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_10",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[3], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_20",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[4], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_50",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[5], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_100",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[6], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_200",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[7], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_1000",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[8], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_overflow",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[9], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_count",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getCount(), cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_sum",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getSum(), cluster, namespace);
- metric(stream, cluster, namespace, "pulsar_in_bytes_total", stats.bytesInCounter);
- metric(stream, cluster, namespace, "pulsar_in_messages_total", stats.msgInCounter);
- metric(stream, cluster, namespace, "pulsar_out_bytes_total", stats.bytesOutCounter);
- metric(stream, cluster, namespace, "pulsar_out_messages_total", stats.msgOutCounter);
+ stats.managedLedgerStats.entrySizeBuckets.refresh();
+ writeMetric(allMetrics, "pulsar_entry_size_le_128", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[0],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_512", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[1],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_1_kb", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[2],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_2_kb", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[3],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_4_kb", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[4],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_16_kb", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[5],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_100_kb",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[6], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_1_mb", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[7],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_overflow",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[8], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_count",
+ stats.managedLedgerStats.entrySizeBuckets.getCount(), cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_sum",
+ stats.managedLedgerStats.entrySizeBuckets.getSum(), cluster, namespace);
+
+ writeReplicationStat(allMetrics, "pulsar_replication_rate_in", stats,
+ replStats -> replStats.msgRateIn, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_rate_out", stats,
+ replStats -> replStats.msgRateOut, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_throughput_in", stats,
+ replStats -> replStats.msgThroughputIn, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_throughput_out", stats,
+ replStats -> replStats.msgThroughputOut, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_backlog", stats,
+ replStats -> replStats.replicationBacklog, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_connected_count", stats,
+ replStats -> replStats.connectedCount, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_rate_expired", stats,
+ replStats -> replStats.msgRateExpired, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_delay_in_seconds", stats,
+ replStats -> replStats.replicationDelayInSeconds, cluster, namespace);
+ }
- metric(stream, cluster, namespace, "pulsar_storage_size", stats.managedLedgerStats.storageSize);
- metric(stream, cluster, namespace, "pulsar_storage_logical_size", stats.managedLedgerStats.storageLogicalSize);
- metric(stream, cluster, namespace, "pulsar_storage_backlog_size", stats.managedLedgerStats.backlogSize);
- metric(stream, cluster, namespace, "pulsar_storage_offloaded_size",
- stats.managedLedgerStats.offloadedStorageUsed);
+ private static void writeMetricWithBrokerDefault(Map<String, ByteBuf> allMetrics, String metricName, Number value,
+ String cluster, String namespace) {
+ ByteBuf buffer = writeGaugeTypeWithBrokerDefault(allMetrics, metricName, cluster);
+ writeSample(buffer, metricName, value, "cluster", cluster, "namespace", namespace);
+ }
- metric(stream, cluster, namespace, "pulsar_storage_write_rate", stats.managedLedgerStats.storageWriteRate);
- metric(stream, cluster, namespace, "pulsar_storage_read_rate", stats.managedLedgerStats.storageReadRate);
+ private static void writePulsarMsgBacklog(Map<String, ByteBuf> allMetrics, Number value,
+ String cluster, String namespace) {
+ ByteBuf buffer = writeGaugeTypeWithBrokerDefault(allMetrics, "pulsar_msg_backlog", cluster);
+ writeSample(buffer, "pulsar_msg_backlog", value, "cluster", cluster, "namespace", namespace, "remote_cluster",
+ "local");
+ }
- metric(stream, cluster, namespace, "pulsar_subscription_delayed", stats.msgDelayed);
+ private static void writeMetric(Map<String, ByteBuf> allMetrics, String metricName, Number value, String cluster,
+ String namespace) {
+ ByteBuf buffer = writeGaugeType(allMetrics, metricName);
+ writeSample(buffer, metricName, value, "cluster", cluster, "namespace", namespace);
+ }
- metricWithRemoteCluster(stream, cluster, namespace, "pulsar_msg_backlog", "local", stats.msgBacklog);
+ private static void writeReplicationStat(Map<String, ByteBuf> allMetrics, String metricName,
+ AggregatedNamespaceStats namespaceStats,
+ Function<AggregatedReplicationStats, Number> replStatsFunction,
Review Comment:
`replStatsFunction` --> `sampleValueFunction`
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/metrics/PrometheusTextFormatUtil.java:
##########
@@ -141,29 +140,32 @@ private static void writeSum(Writer w, DataSketchesOpStatsLogger opStat, String
.append(Double.toString(opStat.getSum(success))).append('\n');
}
- public static void writeMetricsCollectedByPrometheusClient(Writer w, CollectorRegistry registry)
- throws IOException {
- Enumeration<MetricFamilySamples> metricFamilySamples = registry.metricFamilySamples();
- while (metricFamilySamples.hasMoreElements()) {
- MetricFamilySamples metricFamily = metricFamilySamples.nextElement();
-
- for (int i = 0; i < metricFamily.samples.size(); i++) {
- Sample sample = metricFamily.samples.get(i);
- w.write(sample.name);
- w.write('{');
- for (int j = 0; j < sample.labelNames.size(); j++) {
- if (j != 0) {
- w.write(", ");
+ public static void writeMetrics(SimpleTextOutputStream stream, Collection<MetricFamilySamples> familySamples) {
+ for (MetricFamilySamples familySample : familySamples) {
+ stream.write("# TYPE ");
+ stream.write(familySample.name);
+ stream.write(' ');
+ stream.write(familySample.type.name().toLowerCase());
+ stream.write('\n');
+ for (Collector.MetricFamilySamples.Sample sample : familySample.samples) {
+ stream.write(sample.name);
+ if (sample.labelNames.size() > 0) {
+ stream.write('{');
+ for (int i = 0; i < sample.labelNames.size(); ++i) {
+ stream.write(sample.labelNames.get(i));
+ stream.write("=\"");
+ stream.write(sample.labelValues.get(i));
+ stream.write("\",");
Review Comment:
you will write an extra ',' for the last sample
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/metrics/PrometheusTextFormatUtil.java:
##########
@@ -141,29 +140,32 @@ private static void writeSum(Writer w, DataSketchesOpStatsLogger opStat, String
.append(Double.toString(opStat.getSum(success))).append('\n');
}
- public static void writeMetricsCollectedByPrometheusClient(Writer w, CollectorRegistry registry)
- throws IOException {
- Enumeration<MetricFamilySamples> metricFamilySamples = registry.metricFamilySamples();
- while (metricFamilySamples.hasMoreElements()) {
- MetricFamilySamples metricFamily = metricFamilySamples.nextElement();
-
- for (int i = 0; i < metricFamily.samples.size(); i++) {
- Sample sample = metricFamily.samples.get(i);
- w.write(sample.name);
- w.write('{');
- for (int j = 0; j < sample.labelNames.size(); j++) {
- if (j != 0) {
- w.write(", ");
+ public static void writeMetrics(SimpleTextOutputStream stream, Collection<MetricFamilySamples> familySamples) {
+ for (MetricFamilySamples familySample : familySamples) {
+ stream.write("# TYPE ");
+ stream.write(familySample.name);
+ stream.write(' ');
+ stream.write(familySample.type.name().toLowerCase());
+ stream.write('\n');
+ for (Collector.MetricFamilySamples.Sample sample : familySample.samples) {
+ stream.write(sample.name);
+ if (sample.labelNames.size() > 0) {
+ stream.write('{');
+ for (int i = 0; i < sample.labelNames.size(); ++i) {
+ stream.write(sample.labelNames.get(i));
+ stream.write("=\"");
+ stream.write(sample.labelValues.get(i));
+ stream.write("\",");
}
- w.write(sample.labelNames.get(j));
- w.write("=\"");
- w.write(sample.labelValues.get(j));
- w.write('"');
+ stream.write('}');
}
-
- w.write("} ");
- w.write(Collector.doubleToGoString(sample.value));
- w.write('\n');
+ stream.write(' ');
+ stream.write(Collector.doubleToGoString(sample.value));
+ if (sample.timestampMs != null) {
Review Comment:
@hangc0276 What was your motivation not writing the timestamp?
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/NamespaceStatsAggregator.java:
##########
@@ -296,161 +306,264 @@ private static void getTopicStats(Topic topic, TopicStats stats, boolean include
});
}
- private static void printDefaultBrokerStats(SimpleTextOutputStream stream, String cluster) {
- // Print metrics with 0 values. This is necessary to have the available brokers being
- // reported in the brokers dashboard even if they don't have any topic or traffic
- metric(stream, cluster, "pulsar_topics_count", 0);
- metric(stream, cluster, "pulsar_subscriptions_count", 0);
- metric(stream, cluster, "pulsar_producers_count", 0);
- metric(stream, cluster, "pulsar_consumers_count", 0);
- metric(stream, cluster, "pulsar_rate_in", 0);
- metric(stream, cluster, "pulsar_rate_out", 0);
- metric(stream, cluster, "pulsar_throughput_in", 0);
- metric(stream, cluster, "pulsar_throughput_out", 0);
- metric(stream, cluster, "pulsar_storage_size", 0);
- metric(stream, cluster, "pulsar_storage_logical_size", 0);
- metric(stream, cluster, "pulsar_storage_write_rate", 0);
- metric(stream, cluster, "pulsar_storage_read_rate", 0);
- metric(stream, cluster, "pulsar_msg_backlog", 0);
+ private static void printTopicsCountStats(Map<String, ByteBuf> allMetrics, Map<String, Long> namespaceTopicsCount,
+ String cluster) {
+ namespaceTopicsCount.forEach((ns, topicCount) ->
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_topics_count", topicCount, cluster, ns)
Review Comment:
Per my comment from above regarding `MetricsStreams`, I would keep `printDefaultBrokerStats` just add an if per line there.
`if metricStreams.notContains(metricName) metricStreams.writeSample(metricsName, ...)`
We execute it at the end of all metric printing here.
With the original comment, it's easier to figure out we're adding a 0 value in broker level for this metric to be visible in the dashboard.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/metrics/PrometheusTextFormatUtil.java:
##########
@@ -141,29 +140,32 @@ private static void writeSum(Writer w, DataSketchesOpStatsLogger opStat, String
.append(Double.toString(opStat.getSum(success))).append('\n');
}
- public static void writeMetricsCollectedByPrometheusClient(Writer w, CollectorRegistry registry)
- throws IOException {
- Enumeration<MetricFamilySamples> metricFamilySamples = registry.metricFamilySamples();
- while (metricFamilySamples.hasMoreElements()) {
- MetricFamilySamples metricFamily = metricFamilySamples.nextElement();
-
- for (int i = 0; i < metricFamily.samples.size(); i++) {
- Sample sample = metricFamily.samples.get(i);
- w.write(sample.name);
- w.write('{');
- for (int j = 0; j < sample.labelNames.size(); j++) {
- if (j != 0) {
- w.write(", ");
+ public static void writeMetrics(SimpleTextOutputStream stream, Collection<MetricFamilySamples> familySamples) {
+ for (MetricFamilySamples familySample : familySamples) {
+ stream.write("# TYPE ");
+ stream.write(familySample.name);
+ stream.write(' ');
+ stream.write(familySample.type.name().toLowerCase());
+ stream.write('\n');
+ for (Collector.MetricFamilySamples.Sample sample : familySample.samples) {
+ stream.write(sample.name);
+ if (sample.labelNames.size() > 0) {
+ stream.write('{');
+ for (int i = 0; i < sample.labelNames.size(); ++i) {
+ stream.write(sample.labelNames.get(i));
+ stream.write("=\"");
+ stream.write(sample.labelValues.get(i));
+ stream.write("\",");
}
- w.write(sample.labelNames.get(j));
- w.write("=\"");
- w.write(sample.labelValues.get(j));
- w.write('"');
+ stream.write('}');
}
-
- w.write("} ");
- w.write(Collector.doubleToGoString(sample.value));
- w.write('\n');
+ stream.write(' ');
+ stream.write(Collector.doubleToGoString(sample.value));
+ if (sample.timestampMs != null) {
+ stream.write(' ');
+ stream.write(sample.timestampMs.toString());
Review Comment:
Let's avoid `toString()`
Use `stream.write(sample.timestampMs)`
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/NamespaceStatsAggregator.java:
##########
@@ -296,161 +306,264 @@ private static void getTopicStats(Topic topic, TopicStats stats, boolean include
});
}
- private static void printDefaultBrokerStats(SimpleTextOutputStream stream, String cluster) {
- // Print metrics with 0 values. This is necessary to have the available brokers being
- // reported in the brokers dashboard even if they don't have any topic or traffic
- metric(stream, cluster, "pulsar_topics_count", 0);
- metric(stream, cluster, "pulsar_subscriptions_count", 0);
- metric(stream, cluster, "pulsar_producers_count", 0);
- metric(stream, cluster, "pulsar_consumers_count", 0);
- metric(stream, cluster, "pulsar_rate_in", 0);
- metric(stream, cluster, "pulsar_rate_out", 0);
- metric(stream, cluster, "pulsar_throughput_in", 0);
- metric(stream, cluster, "pulsar_throughput_out", 0);
- metric(stream, cluster, "pulsar_storage_size", 0);
- metric(stream, cluster, "pulsar_storage_logical_size", 0);
- metric(stream, cluster, "pulsar_storage_write_rate", 0);
- metric(stream, cluster, "pulsar_storage_read_rate", 0);
- metric(stream, cluster, "pulsar_msg_backlog", 0);
+ private static void printTopicsCountStats(Map<String, ByteBuf> allMetrics, Map<String, Long> namespaceTopicsCount,
+ String cluster) {
+ namespaceTopicsCount.forEach((ns, topicCount) ->
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_topics_count", topicCount, cluster, ns)
+ );
}
- private static void printTopicsCountStats(SimpleTextOutputStream stream, String cluster, String namespace,
- LongAdder topicsCount) {
- metric(stream, cluster, namespace, "pulsar_topics_count", topicsCount.sum());
- }
+ private static void printNamespaceStats(Map<String, ByteBuf> allMetrics, AggregatedNamespaceStats stats,
+ String cluster, String namespace) {
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_topics_count", stats.topicsCount, cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_subscriptions_count", stats.subscriptionsCount, cluster,
+ namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_producers_count", stats.producersCount, cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_consumers_count", stats.consumersCount, cluster, namespace);
+
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_rate_in", stats.rateIn, cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_rate_out", stats.rateOut, cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_throughput_in", stats.throughputIn, cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_throughput_out", stats.throughputOut, cluster, namespace);
+ writeMetric(allMetrics, "pulsar_consumer_msg_ack_rate", stats.messageAckRate, cluster, namespace);
+
+ writeMetric(allMetrics, "pulsar_in_bytes_total", stats.bytesInCounter, cluster, namespace);
+ writeMetric(allMetrics, "pulsar_in_messages_total", stats.msgInCounter, cluster, namespace);
+ writeMetric(allMetrics, "pulsar_out_bytes_total", stats.bytesOutCounter, cluster, namespace);
+ writeMetric(allMetrics, "pulsar_out_messages_total", stats.msgOutCounter, cluster, namespace);
+
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_storage_size", stats.managedLedgerStats.storageSize, cluster,
+ namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_storage_logical_size",
+ stats.managedLedgerStats.storageLogicalSize, cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_backlog_size", stats.managedLedgerStats.backlogSize, cluster,
+ namespace);
+ writeMetric(allMetrics, "pulsar_storage_offloaded_size",
+ stats.managedLedgerStats.offloadedStorageUsed, cluster, namespace);
+
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_storage_write_rate", stats.managedLedgerStats.storageWriteRate,
+ cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_storage_read_rate", stats.managedLedgerStats.storageReadRate,
+ cluster, namespace);
+
+ writeMetric(allMetrics, "pulsar_subscription_delayed", stats.msgDelayed, cluster, namespace);
+
+ writePulsarMsgBacklog(allMetrics, stats.msgBacklog, cluster, namespace);
- private static void printNamespaceStats(SimpleTextOutputStream stream, String cluster, String namespace,
- AggregatedNamespaceStats stats) {
- metric(stream, cluster, namespace, "pulsar_topics_count", stats.topicsCount);
- metric(stream, cluster, namespace, "pulsar_subscriptions_count", stats.subscriptionsCount);
- metric(stream, cluster, namespace, "pulsar_producers_count", stats.producersCount);
- metric(stream, cluster, namespace, "pulsar_consumers_count", stats.consumersCount);
+ stats.managedLedgerStats.storageWriteLatencyBuckets.refresh();
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_0_5",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[0], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_1",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[1], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_5",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[2], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_10",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[3], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_20",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[4], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_50",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[5], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_100",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[6], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_200",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[7], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_1000",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[8], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_overflow",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[9], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_count",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getCount(), cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_sum",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getSum(), cluster, namespace);
- metric(stream, cluster, namespace, "pulsar_rate_in", stats.rateIn);
- metric(stream, cluster, namespace, "pulsar_rate_out", stats.rateOut);
- metric(stream, cluster, namespace, "pulsar_throughput_in", stats.throughputIn);
- metric(stream, cluster, namespace, "pulsar_throughput_out", stats.throughputOut);
- metric(stream, cluster, namespace, "pulsar_consumer_msg_ack_rate", stats.messageAckRate);
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.refresh();
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_0_5",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[0], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_1",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[1], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_5",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[2], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_10",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[3], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_20",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[4], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_50",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[5], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_100",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[6], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_200",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[7], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_1000",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[8], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_overflow",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[9], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_count",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getCount(), cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_sum",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getSum(), cluster, namespace);
- metric(stream, cluster, namespace, "pulsar_in_bytes_total", stats.bytesInCounter);
- metric(stream, cluster, namespace, "pulsar_in_messages_total", stats.msgInCounter);
- metric(stream, cluster, namespace, "pulsar_out_bytes_total", stats.bytesOutCounter);
- metric(stream, cluster, namespace, "pulsar_out_messages_total", stats.msgOutCounter);
+ stats.managedLedgerStats.entrySizeBuckets.refresh();
+ writeMetric(allMetrics, "pulsar_entry_size_le_128", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[0],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_512", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[1],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_1_kb", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[2],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_2_kb", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[3],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_4_kb", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[4],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_16_kb", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[5],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_100_kb",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[6], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_1_mb", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[7],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_overflow",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[8], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_count",
+ stats.managedLedgerStats.entrySizeBuckets.getCount(), cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_sum",
+ stats.managedLedgerStats.entrySizeBuckets.getSum(), cluster, namespace);
+
+ writeReplicationStat(allMetrics, "pulsar_replication_rate_in", stats,
+ replStats -> replStats.msgRateIn, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_rate_out", stats,
+ replStats -> replStats.msgRateOut, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_throughput_in", stats,
+ replStats -> replStats.msgThroughputIn, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_throughput_out", stats,
+ replStats -> replStats.msgThroughputOut, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_backlog", stats,
+ replStats -> replStats.replicationBacklog, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_connected_count", stats,
+ replStats -> replStats.connectedCount, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_rate_expired", stats,
+ replStats -> replStats.msgRateExpired, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_delay_in_seconds", stats,
+ replStats -> replStats.replicationDelayInSeconds, cluster, namespace);
+ }
- metric(stream, cluster, namespace, "pulsar_storage_size", stats.managedLedgerStats.storageSize);
- metric(stream, cluster, namespace, "pulsar_storage_logical_size", stats.managedLedgerStats.storageLogicalSize);
- metric(stream, cluster, namespace, "pulsar_storage_backlog_size", stats.managedLedgerStats.backlogSize);
- metric(stream, cluster, namespace, "pulsar_storage_offloaded_size",
- stats.managedLedgerStats.offloadedStorageUsed);
+ private static void writeMetricWithBrokerDefault(Map<String, ByteBuf> allMetrics, String metricName, Number value,
+ String cluster, String namespace) {
+ ByteBuf buffer = writeGaugeTypeWithBrokerDefault(allMetrics, metricName, cluster);
+ writeSample(buffer, metricName, value, "cluster", cluster, "namespace", namespace);
+ }
- metric(stream, cluster, namespace, "pulsar_storage_write_rate", stats.managedLedgerStats.storageWriteRate);
- metric(stream, cluster, namespace, "pulsar_storage_read_rate", stats.managedLedgerStats.storageReadRate);
+ private static void writePulsarMsgBacklog(Map<String, ByteBuf> allMetrics, Number value,
+ String cluster, String namespace) {
+ ByteBuf buffer = writeGaugeTypeWithBrokerDefault(allMetrics, "pulsar_msg_backlog", cluster);
+ writeSample(buffer, "pulsar_msg_backlog", value, "cluster", cluster, "namespace", namespace, "remote_cluster",
+ "local");
+ }
- metric(stream, cluster, namespace, "pulsar_subscription_delayed", stats.msgDelayed);
+ private static void writeMetric(Map<String, ByteBuf> allMetrics, String metricName, Number value, String cluster,
+ String namespace) {
+ ByteBuf buffer = writeGaugeType(allMetrics, metricName);
+ writeSample(buffer, metricName, value, "cluster", cluster, "namespace", namespace);
+ }
- metricWithRemoteCluster(stream, cluster, namespace, "pulsar_msg_backlog", "local", stats.msgBacklog);
+ private static void writeReplicationStat(Map<String, ByteBuf> allMetrics, String metricName,
+ AggregatedNamespaceStats namespaceStats,
+ Function<AggregatedReplicationStats, Number> replStatsFunction,
+ String cluster, String namespace) {
+ if (!namespaceStats.replicationStats.isEmpty()) {
+ ByteBuf buffer = writeGaugeType(allMetrics, metricName);
+ namespaceStats.replicationStats.forEach((remoteCluster, replStats) ->
+ writeSample(buffer, metricName, replStatsFunction.apply(replStats),
+ "cluster", cluster,
+ "namespace", namespace,
+ "remote_cluster", remoteCluster)
+ );
+ }
+ }
- stats.managedLedgerStats.storageWriteLatencyBuckets.refresh();
- long[] latencyBuckets = stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets();
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_0_5", latencyBuckets[0]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_1", latencyBuckets[1]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_5", latencyBuckets[2]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_10", latencyBuckets[3]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_20", latencyBuckets[4]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_50", latencyBuckets[5]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_100", latencyBuckets[6]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_200", latencyBuckets[7]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_1000", latencyBuckets[8]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_overflow", latencyBuckets[9]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_count",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getCount());
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_sum",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getSum());
+ static ByteBuf writeGaugeType(Map<String, ByteBuf> allMetrics, String metricName) {
Review Comment:
`initGaugeType`
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/NamespaceStatsAggregator.java:
##########
@@ -296,161 +306,264 @@ private static void getTopicStats(Topic topic, TopicStats stats, boolean include
});
}
- private static void printDefaultBrokerStats(SimpleTextOutputStream stream, String cluster) {
- // Print metrics with 0 values. This is necessary to have the available brokers being
- // reported in the brokers dashboard even if they don't have any topic or traffic
- metric(stream, cluster, "pulsar_topics_count", 0);
- metric(stream, cluster, "pulsar_subscriptions_count", 0);
- metric(stream, cluster, "pulsar_producers_count", 0);
- metric(stream, cluster, "pulsar_consumers_count", 0);
- metric(stream, cluster, "pulsar_rate_in", 0);
- metric(stream, cluster, "pulsar_rate_out", 0);
- metric(stream, cluster, "pulsar_throughput_in", 0);
- metric(stream, cluster, "pulsar_throughput_out", 0);
- metric(stream, cluster, "pulsar_storage_size", 0);
- metric(stream, cluster, "pulsar_storage_logical_size", 0);
- metric(stream, cluster, "pulsar_storage_write_rate", 0);
- metric(stream, cluster, "pulsar_storage_read_rate", 0);
- metric(stream, cluster, "pulsar_msg_backlog", 0);
+ private static void printTopicsCountStats(Map<String, ByteBuf> allMetrics, Map<String, Long> namespaceTopicsCount,
+ String cluster) {
+ namespaceTopicsCount.forEach((ns, topicCount) ->
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_topics_count", topicCount, cluster, ns)
+ );
}
- private static void printTopicsCountStats(SimpleTextOutputStream stream, String cluster, String namespace,
- LongAdder topicsCount) {
- metric(stream, cluster, namespace, "pulsar_topics_count", topicsCount.sum());
- }
+ private static void printNamespaceStats(Map<String, ByteBuf> allMetrics, AggregatedNamespaceStats stats,
+ String cluster, String namespace) {
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_topics_count", stats.topicsCount, cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_subscriptions_count", stats.subscriptionsCount, cluster,
+ namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_producers_count", stats.producersCount, cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_consumers_count", stats.consumersCount, cluster, namespace);
+
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_rate_in", stats.rateIn, cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_rate_out", stats.rateOut, cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_throughput_in", stats.throughputIn, cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_throughput_out", stats.throughputOut, cluster, namespace);
+ writeMetric(allMetrics, "pulsar_consumer_msg_ack_rate", stats.messageAckRate, cluster, namespace);
+
+ writeMetric(allMetrics, "pulsar_in_bytes_total", stats.bytesInCounter, cluster, namespace);
+ writeMetric(allMetrics, "pulsar_in_messages_total", stats.msgInCounter, cluster, namespace);
+ writeMetric(allMetrics, "pulsar_out_bytes_total", stats.bytesOutCounter, cluster, namespace);
+ writeMetric(allMetrics, "pulsar_out_messages_total", stats.msgOutCounter, cluster, namespace);
+
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_storage_size", stats.managedLedgerStats.storageSize, cluster,
+ namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_storage_logical_size",
+ stats.managedLedgerStats.storageLogicalSize, cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_backlog_size", stats.managedLedgerStats.backlogSize, cluster,
+ namespace);
+ writeMetric(allMetrics, "pulsar_storage_offloaded_size",
+ stats.managedLedgerStats.offloadedStorageUsed, cluster, namespace);
+
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_storage_write_rate", stats.managedLedgerStats.storageWriteRate,
+ cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_storage_read_rate", stats.managedLedgerStats.storageReadRate,
+ cluster, namespace);
+
+ writeMetric(allMetrics, "pulsar_subscription_delayed", stats.msgDelayed, cluster, namespace);
+
+ writePulsarMsgBacklog(allMetrics, stats.msgBacklog, cluster, namespace);
- private static void printNamespaceStats(SimpleTextOutputStream stream, String cluster, String namespace,
- AggregatedNamespaceStats stats) {
- metric(stream, cluster, namespace, "pulsar_topics_count", stats.topicsCount);
- metric(stream, cluster, namespace, "pulsar_subscriptions_count", stats.subscriptionsCount);
- metric(stream, cluster, namespace, "pulsar_producers_count", stats.producersCount);
- metric(stream, cluster, namespace, "pulsar_consumers_count", stats.consumersCount);
+ stats.managedLedgerStats.storageWriteLatencyBuckets.refresh();
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_0_5",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[0], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_1",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[1], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_5",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[2], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_10",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[3], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_20",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[4], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_50",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[5], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_100",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[6], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_200",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[7], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_1000",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[8], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_overflow",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[9], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_count",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getCount(), cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_sum",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getSum(), cluster, namespace);
- metric(stream, cluster, namespace, "pulsar_rate_in", stats.rateIn);
- metric(stream, cluster, namespace, "pulsar_rate_out", stats.rateOut);
- metric(stream, cluster, namespace, "pulsar_throughput_in", stats.throughputIn);
- metric(stream, cluster, namespace, "pulsar_throughput_out", stats.throughputOut);
- metric(stream, cluster, namespace, "pulsar_consumer_msg_ack_rate", stats.messageAckRate);
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.refresh();
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_0_5",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[0], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_1",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[1], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_5",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[2], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_10",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[3], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_20",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[4], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_50",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[5], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_100",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[6], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_200",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[7], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_1000",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[8], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_overflow",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[9], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_count",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getCount(), cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_sum",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getSum(), cluster, namespace);
- metric(stream, cluster, namespace, "pulsar_in_bytes_total", stats.bytesInCounter);
- metric(stream, cluster, namespace, "pulsar_in_messages_total", stats.msgInCounter);
- metric(stream, cluster, namespace, "pulsar_out_bytes_total", stats.bytesOutCounter);
- metric(stream, cluster, namespace, "pulsar_out_messages_total", stats.msgOutCounter);
+ stats.managedLedgerStats.entrySizeBuckets.refresh();
+ writeMetric(allMetrics, "pulsar_entry_size_le_128", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[0],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_512", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[1],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_1_kb", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[2],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_2_kb", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[3],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_4_kb", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[4],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_16_kb", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[5],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_100_kb",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[6], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_1_mb", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[7],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_overflow",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[8], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_count",
+ stats.managedLedgerStats.entrySizeBuckets.getCount(), cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_sum",
+ stats.managedLedgerStats.entrySizeBuckets.getSum(), cluster, namespace);
+
+ writeReplicationStat(allMetrics, "pulsar_replication_rate_in", stats,
+ replStats -> replStats.msgRateIn, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_rate_out", stats,
+ replStats -> replStats.msgRateOut, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_throughput_in", stats,
+ replStats -> replStats.msgThroughputIn, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_throughput_out", stats,
+ replStats -> replStats.msgThroughputOut, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_backlog", stats,
+ replStats -> replStats.replicationBacklog, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_connected_count", stats,
+ replStats -> replStats.connectedCount, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_rate_expired", stats,
+ replStats -> replStats.msgRateExpired, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_delay_in_seconds", stats,
+ replStats -> replStats.replicationDelayInSeconds, cluster, namespace);
+ }
- metric(stream, cluster, namespace, "pulsar_storage_size", stats.managedLedgerStats.storageSize);
- metric(stream, cluster, namespace, "pulsar_storage_logical_size", stats.managedLedgerStats.storageLogicalSize);
- metric(stream, cluster, namespace, "pulsar_storage_backlog_size", stats.managedLedgerStats.backlogSize);
- metric(stream, cluster, namespace, "pulsar_storage_offloaded_size",
- stats.managedLedgerStats.offloadedStorageUsed);
+ private static void writeMetricWithBrokerDefault(Map<String, ByteBuf> allMetrics, String metricName, Number value,
+ String cluster, String namespace) {
+ ByteBuf buffer = writeGaugeTypeWithBrokerDefault(allMetrics, metricName, cluster);
+ writeSample(buffer, metricName, value, "cluster", cluster, "namespace", namespace);
+ }
- metric(stream, cluster, namespace, "pulsar_storage_write_rate", stats.managedLedgerStats.storageWriteRate);
- metric(stream, cluster, namespace, "pulsar_storage_read_rate", stats.managedLedgerStats.storageReadRate);
+ private static void writePulsarMsgBacklog(Map<String, ByteBuf> allMetrics, Number value,
+ String cluster, String namespace) {
+ ByteBuf buffer = writeGaugeTypeWithBrokerDefault(allMetrics, "pulsar_msg_backlog", cluster);
+ writeSample(buffer, "pulsar_msg_backlog", value, "cluster", cluster, "namespace", namespace, "remote_cluster",
+ "local");
+ }
- metric(stream, cluster, namespace, "pulsar_subscription_delayed", stats.msgDelayed);
+ private static void writeMetric(Map<String, ByteBuf> allMetrics, String metricName, Number value, String cluster,
+ String namespace) {
+ ByteBuf buffer = writeGaugeType(allMetrics, metricName);
+ writeSample(buffer, metricName, value, "cluster", cluster, "namespace", namespace);
+ }
- metricWithRemoteCluster(stream, cluster, namespace, "pulsar_msg_backlog", "local", stats.msgBacklog);
+ private static void writeReplicationStat(Map<String, ByteBuf> allMetrics, String metricName,
+ AggregatedNamespaceStats namespaceStats,
+ Function<AggregatedReplicationStats, Number> replStatsFunction,
+ String cluster, String namespace) {
+ if (!namespaceStats.replicationStats.isEmpty()) {
+ ByteBuf buffer = writeGaugeType(allMetrics, metricName);
+ namespaceStats.replicationStats.forEach((remoteCluster, replStats) ->
+ writeSample(buffer, metricName, replStatsFunction.apply(replStats),
+ "cluster", cluster,
+ "namespace", namespace,
+ "remote_cluster", remoteCluster)
+ );
+ }
+ }
- stats.managedLedgerStats.storageWriteLatencyBuckets.refresh();
- long[] latencyBuckets = stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets();
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_0_5", latencyBuckets[0]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_1", latencyBuckets[1]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_5", latencyBuckets[2]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_10", latencyBuckets[3]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_20", latencyBuckets[4]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_50", latencyBuckets[5]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_100", latencyBuckets[6]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_200", latencyBuckets[7]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_1000", latencyBuckets[8]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_overflow", latencyBuckets[9]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_count",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getCount());
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_sum",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getSum());
+ static ByteBuf writeGaugeType(Map<String, ByteBuf> allMetrics, String metricName) {
+ if (!allMetrics.containsKey(metricName)) {
+ ByteBuf buffer = UnpooledByteBufAllocator.DEFAULT.compositeDirectBuffer(MAX_COMPONENTS);
Review Comment:
@merlimat - not sure about this
##########
pulsar-common/src/main/java/org/apache/pulsar/common/util/SimpleTextOutputStream.java:
##########
@@ -131,4 +131,9 @@ public SimpleTextOutputStream write(double d) {
write(r);
return this;
}
+
+ public void write(ByteBuf byteBuf) {
+ buffer.writeBytes(byteBuf);
+ byteBuf.release();
Review Comment:
That's very dangerous. Always keep the release where you allocated
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/TopicStats.java:
##########
@@ -102,380 +104,401 @@ public void reset() {
compactionLatencyBuckets.reset();
}
- static void resetTypes() {
- metricWithTypeDefinition.clear();
- }
-
- static void printTopicStats(SimpleTextOutputStream stream, String cluster, String namespace, String topic,
- TopicStats stats, Optional<CompactorMXBean> compactorMXBean,
- boolean splitTopicAndPartitionIndexLabel) {
- metric(stream, cluster, namespace, topic, "pulsar_subscriptions_count", stats.subscriptionsCount,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_producers_count", stats.producersCount,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_consumers_count", stats.consumersCount,
- splitTopicAndPartitionIndexLabel);
-
- metric(stream, cluster, namespace, topic, "pulsar_rate_in", stats.rateIn,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_rate_out", stats.rateOut,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_throughput_in", stats.throughputIn,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_throughput_out", stats.throughputOut,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_average_msg_size", stats.averageMsgSize,
- splitTopicAndPartitionIndexLabel);
-
- metric(stream, cluster, namespace, topic, "pulsar_storage_size", stats.managedLedgerStats.storageSize,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_logical_size",
- stats.managedLedgerStats.storageLogicalSize, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_msg_backlog", stats.msgBacklog,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_backlog_size",
- stats.managedLedgerStats.backlogSize, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_publish_rate_limit_times", stats.publishRateLimitedTimes,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_offloaded_size", stats.managedLedgerStats
- .offloadedStorageUsed, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_backlog_quota_limit", stats.backlogQuotaLimit,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_backlog_quota_limit_time",
- stats.backlogQuotaLimitTime, splitTopicAndPartitionIndexLabel);
-
- long[] latencyBuckets = stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets();
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_le_0_5", latencyBuckets[0],
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_le_1", latencyBuckets[1],
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_le_5", latencyBuckets[2],
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_le_10", latencyBuckets[3],
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_le_20", latencyBuckets[4],
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_le_50", latencyBuckets[5],
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_le_100", latencyBuckets[6],
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_le_200", latencyBuckets[7],
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_le_1000", latencyBuckets[8],
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_overflow", latencyBuckets[9],
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_count",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getCount(), splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_sum",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getSum(), splitTopicAndPartitionIndexLabel);
-
- long[] ledgerWriteLatencyBuckets = stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets();
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_0_5",
- ledgerWriteLatencyBuckets[0], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_1",
- ledgerWriteLatencyBuckets[1], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_5",
- ledgerWriteLatencyBuckets[2], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_10",
- ledgerWriteLatencyBuckets[3], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_20",
- ledgerWriteLatencyBuckets[4], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_50",
- ledgerWriteLatencyBuckets[5], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_100",
- ledgerWriteLatencyBuckets[6], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_200",
- ledgerWriteLatencyBuckets[7], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_1000",
- ledgerWriteLatencyBuckets[8], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_overflow",
- ledgerWriteLatencyBuckets[9], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_count",
+ public static void printTopicStats(Map<String, ByteBuf> allMetrics, TopicStats stats,
+ Optional<CompactorMXBean> compactorMXBean, String cluster, String namespace,
+ String name, boolean splitTopicAndPartitionIndexLabel) {
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_subscriptions_count", stats.subscriptionsCount,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_producers_count", stats.producersCount,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_consumers_count", stats.consumersCount,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_rate_in", stats.rateIn,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_rate_out", stats.rateOut,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_throughput_in", stats.throughputIn,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_throughput_out", stats.throughputOut,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_average_msg_size", stats.averageMsgSize,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_storage_size", stats.managedLedgerStats.storageSize,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_storage_logical_size",
+ stats.managedLedgerStats.storageLogicalSize, cluster, namespace, name,
+ splitTopicAndPartitionIndexLabel);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_msg_backlog", stats.msgBacklog,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_backlog_size", stats.managedLedgerStats.backlogSize,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_publish_rate_limit_times", stats.publishRateLimitedTimes,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_offloaded_size", stats.managedLedgerStats
+ .offloadedStorageUsed, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_backlog_quota_limit", stats.backlogQuotaLimit,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_backlog_quota_limit_time", stats.backlogQuotaLimitTime,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_0_5",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[0],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_1",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[1],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_5",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[2],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_10",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[3],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_20",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[4],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_50",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[5],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_100",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[6],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_200",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[7],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_1000",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[8],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_overflow",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[9],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_count",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getCount(),
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_sum",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getSum(), cluster, namespace, name,
+ splitTopicAndPartitionIndexLabel);
+
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_0_5",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[0],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_1",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[1],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_5",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[2],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_10",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[3],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_20",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[4],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_50",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[5],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_100",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[6],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_200",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[7],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_1000",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[8],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_overflow",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[9],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_count",
stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getCount(),
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_sum",
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_sum",
stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getSum(),
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+ writeMetric(allMetrics, "pulsar_entry_size_le_128",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[0],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_entry_size_le_512",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[1],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_entry_size_le_1_kb",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[2],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_entry_size_le_2_kb",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[3],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_entry_size_le_4_kb",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[4],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_entry_size_le_16_kb",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[5],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_entry_size_le_100_kb",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[6],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_entry_size_le_1_mb",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[7],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_entry_size_le_overflow",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[8],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_entry_size_count", stats.managedLedgerStats.entrySizeBuckets.getCount(),
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_entry_size_sum", stats.managedLedgerStats.entrySizeBuckets.getSum(),
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+ writeProducerStat(allMetrics, "pulsar_producer_msg_rate_in", stats,
+ p -> p.msgRateIn, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeProducerStat(allMetrics, "pulsar_producer_msg_throughput_in", stats,
+ p -> p.msgThroughputIn, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeProducerStat(allMetrics, "pulsar_producer_msg_average_Size", stats,
+ p -> p.averageMsgSize, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_back_log", stats, s -> s.msgBacklog,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_back_log_no_delayed",
+ stats, s -> s.msgBacklogNoDelayed, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_delayed",
+ stats, s -> s.msgDelayed, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_msg_rate_redeliver",
+ stats, s -> s.msgRateRedeliver, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_unacked_messages",
+ stats, s -> s.unackedMessages, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_blocked_on_unacked_messages",
+ stats, s -> s.blockedSubscriptionOnUnackedMsgs ? 1 : 0, cluster, namespace, name,
+ splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_msg_rate_out",
+ stats, s -> s.msgRateOut, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_msg_ack_rate",
+ stats, s -> s.messageAckRate, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_msg_throughput_out",
+ stats, s -> s.msgThroughputOut, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_out_bytes_total",
+ stats, s -> s.bytesOutCounter, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_out_messages_total",
+ stats, s -> s.msgOutCounter, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_last_expire_timestamp",
+ stats, s -> s.lastExpireTimestamp, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_last_acked_timestamp",
+ stats, s -> s.lastAckedTimestamp, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_last_consumed_flow_timestamp",
+ stats, s -> s.lastConsumedFlowTimestamp, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_last_consumed_timestamp",
+ stats, s -> s.lastConsumedTimestamp, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_last_mark_delete_advanced_timestamp",
+ stats, s -> s.lastMarkDeleteAdvancedTimestamp, cluster, namespace, name,
+ splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_msg_rate_expired",
+ stats, s -> s.msgRateExpired, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_total_msg_expired",
+ stats, s -> s.totalMsgExpired, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_msg_drop_rate",
+ stats, s -> s.msgDropRate, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_consumers_count",
+ stats, s -> s.consumersCount, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+ writeConsumerStat(allMetrics, "pulsar_consumer_msg_rate_redeliver", stats, c -> c.msgRateRedeliver,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeConsumerStat(allMetrics, "pulsar_consumer_unacked_messages", stats, c -> c.unackedMessages,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeConsumerStat(allMetrics, "pulsar_consumer_blocked_on_unacked_messages",
+ stats, c -> c.blockedSubscriptionOnUnackedMsgs ? 1 : 0,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeConsumerStat(allMetrics, "pulsar_consumer_msg_rate_out", stats, c -> c.msgRateOut,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+ writeConsumerStat(allMetrics, "pulsar_consumer_msg_ack_rate", stats, c -> c.msgAckRate,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+ writeConsumerStat(allMetrics, "pulsar_consumer_msg_throughput_out", stats, c -> c.msgThroughputOut,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeConsumerStat(allMetrics, "pulsar_consumer_available_permits", stats, c -> c.availablePermits,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeConsumerStat(allMetrics, "pulsar_out_bytes_total", stats, c -> c.bytesOutCounter,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeConsumerStat(allMetrics, "pulsar_out_messages_total", stats, c -> c.msgOutCounter,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+ writeReplicationStat(allMetrics, "pulsar_replication_rate_in", stats, r -> r.msgRateIn,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(allMetrics, "pulsar_replication_rate_out", stats, r -> r.msgRateOut,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(allMetrics, "pulsar_replication_throughput_in", stats, r -> r.msgThroughputIn,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(allMetrics, "pulsar_replication_throughput_out", stats, r -> r.msgThroughputOut,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(allMetrics, "pulsar_replication_backlog", stats, r -> r.replicationBacklog,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(allMetrics, "pulsar_replication_connected_count", stats, r -> r.connectedCount,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(allMetrics, "pulsar_replication_rate_expired", stats, r -> r.msgRateExpired,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(allMetrics, "pulsar_replication_delay_in_seconds", stats,
+ r -> r.replicationDelayInSeconds, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+ writeMetric(allMetrics, "pulsar_in_bytes_total", stats.bytesInCounter, cluster, namespace, name,
+ splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_in_messages_total", stats.msgInCounter, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- long[] entrySizeBuckets = stats.managedLedgerStats.entrySizeBuckets.getBuckets();
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_le_128", entrySizeBuckets[0],
+ // Compaction
+
+ writeCompactionStat(allMetrics, "pulsar_compaction_removed_event_count", stats.compactionRemovedEventCount,
+ compactorMXBean, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(allMetrics, "pulsar_compaction_succeed_count", stats.compactionSucceedCount,
+ compactorMXBean, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(allMetrics, "pulsar_compaction_failed_count", stats.compactionFailedCount, compactorMXBean,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(allMetrics, "pulsar_compaction_duration_time_in_mills", stats.compactionDurationTimeInMills,
+ compactorMXBean, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(allMetrics, "pulsar_compaction_read_throughput", stats.compactionReadThroughput,
+ compactorMXBean, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(allMetrics, "pulsar_compaction_write_throughput", stats.compactionWriteThroughput,
+ compactorMXBean, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(allMetrics, "pulsar_compaction_compacted_entries_count",
+ stats.compactionCompactedEntriesCount, compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_le_512", entrySizeBuckets[1],
+ writeCompactionStat(allMetrics, "pulsar_compaction_compacted_entries_size",
+ stats.compactionCompactedEntriesSize, compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_le_1_kb", entrySizeBuckets[2],
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_le_0_5",
+ stats.compactionLatencyBuckets.getBuckets()[0], compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_le_2_kb", entrySizeBuckets[3],
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_le_1",
+ stats.compactionLatencyBuckets.getBuckets()[1], compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_le_4_kb", entrySizeBuckets[4],
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_le_5",
+ stats.compactionLatencyBuckets.getBuckets()[2], compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_le_16_kb", entrySizeBuckets[5],
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_le_10",
+ stats.compactionLatencyBuckets.getBuckets()[3], compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_le_100_kb", entrySizeBuckets[6],
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_le_20",
+ stats.compactionLatencyBuckets.getBuckets()[4], compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_le_1_mb", entrySizeBuckets[7],
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_le_50",
+ stats.compactionLatencyBuckets.getBuckets()[5], compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_le_overflow", entrySizeBuckets[8],
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_le_100",
+ stats.compactionLatencyBuckets.getBuckets()[6], compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_count",
- stats.managedLedgerStats.entrySizeBuckets.getCount(), splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_sum",
- stats.managedLedgerStats.entrySizeBuckets.getSum(), splitTopicAndPartitionIndexLabel);
-
- stats.producerStats.forEach((p, producerStats) -> {
- metric(stream, cluster, namespace, topic, p, producerStats.producerId, "pulsar_producer_msg_rate_in",
- producerStats.msgRateIn, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, p, producerStats.producerId, "pulsar_producer_msg_throughput_in",
- producerStats.msgThroughputIn, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, p, producerStats.producerId, "pulsar_producer_msg_average_Size",
- producerStats.averageMsgSize, splitTopicAndPartitionIndexLabel);
- });
-
- stats.subscriptionStats.forEach((n, subsStats) -> {
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_back_log",
- subsStats.msgBacklog, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_back_log_no_delayed",
- subsStats.msgBacklogNoDelayed, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_delayed",
- subsStats.msgDelayed, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_msg_rate_redeliver",
- subsStats.msgRateRedeliver, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_unacked_messages",
- subsStats.unackedMessages, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_blocked_on_unacked_messages",
- subsStats.blockedSubscriptionOnUnackedMsgs ? 1 : 0, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_msg_rate_out",
- subsStats.msgRateOut, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_msg_ack_rate",
- subsStats.messageAckRate, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_msg_throughput_out",
- subsStats.msgThroughputOut, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_out_bytes_total",
- subsStats.bytesOutCounter, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_out_messages_total",
- subsStats.msgOutCounter, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_last_expire_timestamp",
- subsStats.lastExpireTimestamp, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_last_acked_timestamp",
- subsStats.lastAckedTimestamp, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_last_consumed_flow_timestamp",
- subsStats.lastConsumedFlowTimestamp, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_last_consumed_timestamp",
- subsStats.lastConsumedTimestamp, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_last_mark_delete_advanced_timestamp",
- subsStats.lastMarkDeleteAdvancedTimestamp, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_msg_rate_expired",
- subsStats.msgRateExpired, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_total_msg_expired",
- subsStats.totalMsgExpired, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_msg_drop_rate",
- subsStats.msgDropRate, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_consumers_count",
- subsStats.consumersCount, splitTopicAndPartitionIndexLabel);
- subsStats.consumerStat.forEach((c, consumerStats) -> {
- metric(stream, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_msg_rate_redeliver", consumerStats.msgRateRedeliver,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_unacked_messages", consumerStats.unackedMessages,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_blocked_on_unacked_messages",
- consumerStats.blockedSubscriptionOnUnackedMsgs ? 1 : 0,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_msg_rate_out", consumerStats.msgRateOut,
- splitTopicAndPartitionIndexLabel);
-
- metric(stream, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_msg_ack_rate", consumerStats.msgAckRate,
- splitTopicAndPartitionIndexLabel);
-
- metric(stream, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_msg_throughput_out", consumerStats.msgThroughputOut,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_available_permits", consumerStats.availablePermits,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_out_bytes_total", consumerStats.bytesOutCounter,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_out_messages_total", consumerStats.msgOutCounter,
- splitTopicAndPartitionIndexLabel);
- });
- });
-
- if (!stats.replicationStats.isEmpty()) {
- stats.replicationStats.forEach((remoteCluster, replStats) -> {
- metricWithRemoteCluster(stream, cluster, namespace, topic, "pulsar_replication_rate_in", remoteCluster,
- replStats.msgRateIn, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(stream, cluster, namespace, topic, "pulsar_replication_rate_out", remoteCluster,
- replStats.msgRateOut, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(stream, cluster, namespace, topic, "pulsar_replication_throughput_in",
- remoteCluster, replStats.msgThroughputIn, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(stream, cluster, namespace, topic, "pulsar_replication_throughput_out",
- remoteCluster, replStats.msgThroughputOut, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(stream, cluster, namespace, topic, "pulsar_replication_backlog", remoteCluster,
- replStats.replicationBacklog, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(stream, cluster, namespace, topic, "pulsar_replication_connected_count",
- remoteCluster, replStats.connectedCount, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(stream, cluster, namespace, topic, "pulsar_replication_rate_expired",
- remoteCluster, replStats.msgRateExpired, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(stream, cluster, namespace, topic, "pulsar_replication_delay_in_seconds",
- remoteCluster, replStats.replicationDelayInSeconds, splitTopicAndPartitionIndexLabel);
- });
- }
-
- metric(stream, cluster, namespace, topic, "pulsar_in_bytes_total", stats.bytesInCounter,
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_le_200",
+ stats.compactionLatencyBuckets.getBuckets()[7], compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_in_messages_total", stats.msgInCounter,
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_le_1000",
+ stats.compactionLatencyBuckets.getBuckets()[8], compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
-
- // Compaction
- boolean hasCompaction = compactorMXBean.flatMap(mxBean -> mxBean.getCompactionRecordForTopic(topic))
- .map(__ -> true).orElse(false);
- if (hasCompaction) {
- metric(stream, cluster, namespace, topic, "pulsar_compaction_removed_event_count",
- stats.compactionRemovedEventCount, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_succeed_count",
- stats.compactionSucceedCount, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_failed_count",
- stats.compactionFailedCount, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_duration_time_in_mills",
- stats.compactionDurationTimeInMills, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_read_throughput",
- stats.compactionReadThroughput, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_write_throughput",
- stats.compactionWriteThroughput, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_compacted_entries_count",
- stats.compactionCompactedEntriesCount, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_compacted_entries_size",
- stats.compactionCompactedEntriesSize, splitTopicAndPartitionIndexLabel);
- long[] compactionLatencyBuckets = stats.compactionLatencyBuckets.getBuckets();
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_le_0_5",
- compactionLatencyBuckets[0], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_le_1",
- compactionLatencyBuckets[1], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_le_5",
- compactionLatencyBuckets[2], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_le_10",
- compactionLatencyBuckets[3], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_le_20",
- compactionLatencyBuckets[4], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_le_50",
- compactionLatencyBuckets[5], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_le_100",
- compactionLatencyBuckets[6], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_le_200",
- compactionLatencyBuckets[7], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_le_1000",
- compactionLatencyBuckets[8], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_overflow",
- compactionLatencyBuckets[9], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_sum",
- stats.compactionLatencyBuckets.getSum(), splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_count",
- stats.compactionLatencyBuckets.getCount(), splitTopicAndPartitionIndexLabel);
- }
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_overflow",
+ stats.compactionLatencyBuckets.getBuckets()[9], compactorMXBean, cluster, namespace, name,
+ splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_sum", stats.compactionLatencyBuckets.getSum(),
+ compactorMXBean, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_count", stats.compactionLatencyBuckets.getCount(),
+ compactorMXBean, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
}
- static void metricType(SimpleTextOutputStream stream, String name) {
-
- if (!metricWithTypeDefinition.containsKey(name)) {
- metricWithTypeDefinition.put(name, "gauge");
- stream.write("# TYPE ").write(name).write(" gauge\n");
- }
-
+ private static void writeMetric(Map<String, ByteBuf> allMetrics, String metricName, Number value, String cluster,
+ String namespace, String topic, boolean splitTopicAndPartitionIndexLabel) {
+ ByteBuf buffer = writeGaugeType(allMetrics, metricName);
+ writeTopicSample(buffer, metricName, value, cluster, namespace, topic, splitTopicAndPartitionIndexLabel);
}
- private static void metric(SimpleTextOutputStream stream, String cluster, String namespace, String topic,
- String name, double value, boolean splitTopicAndPartitionIndexLabel) {
- metricType(stream, name);
- appendRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel).write("\"} ");
- stream.write(value);
- appendEndings(stream);
+ private static void writeProducerStat(Map<String, ByteBuf> allMetrics, String metricName, TopicStats topicStats,
+ Function<AggregatedProducerStats, Number> valueFunction,
+ String cluster, String namespace, String topic,
+ boolean splitTopicAndPartitionIndexLabel) {
+ ByteBuf buffer = writeGaugeType(allMetrics, metricName);
+ topicStats.producerStats.forEach((p, producerStats) ->
+ writeTopicSample(buffer, metricName, valueFunction.apply(producerStats), cluster, namespace, topic,
+ splitTopicAndPartitionIndexLabel, "producer_name", p, "producer_id",
+ String.valueOf(producerStats.producerId)));
}
- private static void metric(SimpleTextOutputStream stream, String cluster, String namespace, String topic,
- String subscription, String name, long value, boolean splitTopicAndPartitionIndexLabel) {
- metricType(stream, name);
- appendRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel)
- .write("\",subscription=\"").write(subscription).write("\"} ");
- stream.write(value);
- appendEndings(stream);
+ private static void writeSubscriptionStat(Map<String, ByteBuf> allMetrics, String metricName, TopicStats topicStats,
+ Function<AggregatedSubscriptionStats, Number> valueFunction,
+ String cluster, String namespace, String topic,
+ boolean splitTopicAndPartitionIndexLabel) {
+ ByteBuf buffer = writeGaugeType(allMetrics, metricName);
+ topicStats.subscriptionStats.forEach((s, subStats) ->
+ writeTopicSample(buffer, metricName, valueFunction.apply(subStats), cluster, namespace, topic,
+ splitTopicAndPartitionIndexLabel, "subscription", s));
}
- private static void metric(SimpleTextOutputStream stream, String cluster, String namespace, String topic,
- String producerName, long produceId, String name, double value, boolean splitTopicAndPartitionIndexLabel) {
- metricType(stream, name);
- appendRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel)
- .write("\",producer_name=\"").write(producerName)
- .write("\",producer_id=\"").write(produceId).write("\"} ");
- stream.write(value);
- appendEndings(stream);
+ private static void writeConsumerStat(Map<String, ByteBuf> allMetrics, String metricName, TopicStats topicStats,
+ Function<AggregatedConsumerStats, Number> valueFunction,
+ String cluster, String namespace, String topic,
+ boolean splitTopicAndPartitionIndexLabel) {
+ ByteBuf buffer = writeGaugeType(allMetrics, metricName);
+ topicStats.subscriptionStats.forEach((s, subStats) ->
+ subStats.consumerStat.forEach((c, conStats) ->
+ writeTopicSample(buffer, metricName, valueFunction.apply(conStats), cluster, namespace, topic,
+ splitTopicAndPartitionIndexLabel, "subscription", s, "consumer_name", c.consumerName(),
+ "consumer_id", String.valueOf(c.consumerId()))
+ ));
}
- private static void metric(SimpleTextOutputStream stream, String cluster, String namespace, String topic,
- String subscription, String name, double value, boolean splitTopicAndPartitionIndexLabel) {
- metricType(stream, name);
- appendRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel)
- .write("\",subscription=\"").write(subscription).write("\"} ");
- stream.write(value);
- appendEndings(stream);
- }
- private static void metric(SimpleTextOutputStream stream, String cluster, String namespace, String topic,
- String subscription, String consumerName, long consumerId, String name, long value,
- boolean splitTopicAndPartitionIndexLabel) {
- metricType(stream, name);
- appendRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel)
- .write("\",subscription=\"").write(subscription)
- .write("\",consumer_name=\"").write(consumerName).write("\",consumer_id=\"").write(consumerId)
- .write("\"} ");
- stream.write(value);
- appendEndings(stream);
+ private static void writeReplicationStat(Map<String, ByteBuf> allMetrics, String metricName, TopicStats topicStats,
+ Function<AggregatedReplicationStats, Number> valueFunction,
+ String cluster, String namespace, String topic,
+ boolean splitTopicAndPartitionIndexLabel) {
+ if (!topicStats.replicationStats.isEmpty()) {
+ ByteBuf buffer = writeGaugeType(allMetrics, metricName);
+ topicStats.replicationStats.forEach((remoteCluster, replStats) ->
+ writeTopicSample(buffer, metricName, valueFunction.apply(replStats), cluster, namespace, topic,
+ splitTopicAndPartitionIndexLabel, "remote_cluster", remoteCluster)
+ );
+ }
}
- private static void metric(SimpleTextOutputStream stream, String cluster, String namespace, String topic,
- String subscription, String consumerName, long consumerId, String name, double value,
- boolean splitTopicAndPartitionIndexLabel) {
- metricType(stream, name);
- appendRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel)
- .write("\",subscription=\"").write(subscription)
- .write("\",consumer_name=\"").write(consumerName).write("\",consumer_id=\"")
- .write(consumerId).write("\"} ");
- stream.write(value);
- appendEndings(stream);
+ private static void writeCompactionStat(Map<String, ByteBuf> allMetrics, String metricName,
+ Number value, Optional<CompactorMXBean> compactorMXBean,
+ String cluster, String namespace, String topic,
+ boolean splitTopicAndPartitionIndexLabel) {
+ boolean hasCompaction = compactorMXBean.flatMap(mxBean -> mxBean.getCompactionRecordForTopic(topic))
Review Comment:
Why push it inside and run it multiple times? I prefer the original way of checking and if true, calling it.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/NamespaceStatsAggregator.java:
##########
@@ -296,161 +306,264 @@ private static void getTopicStats(Topic topic, TopicStats stats, boolean include
});
}
- private static void printDefaultBrokerStats(SimpleTextOutputStream stream, String cluster) {
- // Print metrics with 0 values. This is necessary to have the available brokers being
- // reported in the brokers dashboard even if they don't have any topic or traffic
- metric(stream, cluster, "pulsar_topics_count", 0);
- metric(stream, cluster, "pulsar_subscriptions_count", 0);
- metric(stream, cluster, "pulsar_producers_count", 0);
- metric(stream, cluster, "pulsar_consumers_count", 0);
- metric(stream, cluster, "pulsar_rate_in", 0);
- metric(stream, cluster, "pulsar_rate_out", 0);
- metric(stream, cluster, "pulsar_throughput_in", 0);
- metric(stream, cluster, "pulsar_throughput_out", 0);
- metric(stream, cluster, "pulsar_storage_size", 0);
- metric(stream, cluster, "pulsar_storage_logical_size", 0);
- metric(stream, cluster, "pulsar_storage_write_rate", 0);
- metric(stream, cluster, "pulsar_storage_read_rate", 0);
- metric(stream, cluster, "pulsar_msg_backlog", 0);
+ private static void printTopicsCountStats(Map<String, ByteBuf> allMetrics, Map<String, Long> namespaceTopicsCount,
+ String cluster) {
+ namespaceTopicsCount.forEach((ns, topicCount) ->
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_topics_count", topicCount, cluster, ns)
+ );
}
- private static void printTopicsCountStats(SimpleTextOutputStream stream, String cluster, String namespace,
- LongAdder topicsCount) {
- metric(stream, cluster, namespace, "pulsar_topics_count", topicsCount.sum());
- }
+ private static void printNamespaceStats(Map<String, ByteBuf> allMetrics, AggregatedNamespaceStats stats,
+ String cluster, String namespace) {
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_topics_count", stats.topicsCount, cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_subscriptions_count", stats.subscriptionsCount, cluster,
+ namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_producers_count", stats.producersCount, cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_consumers_count", stats.consumersCount, cluster, namespace);
+
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_rate_in", stats.rateIn, cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_rate_out", stats.rateOut, cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_throughput_in", stats.throughputIn, cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_throughput_out", stats.throughputOut, cluster, namespace);
+ writeMetric(allMetrics, "pulsar_consumer_msg_ack_rate", stats.messageAckRate, cluster, namespace);
+
+ writeMetric(allMetrics, "pulsar_in_bytes_total", stats.bytesInCounter, cluster, namespace);
+ writeMetric(allMetrics, "pulsar_in_messages_total", stats.msgInCounter, cluster, namespace);
+ writeMetric(allMetrics, "pulsar_out_bytes_total", stats.bytesOutCounter, cluster, namespace);
+ writeMetric(allMetrics, "pulsar_out_messages_total", stats.msgOutCounter, cluster, namespace);
+
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_storage_size", stats.managedLedgerStats.storageSize, cluster,
+ namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_storage_logical_size",
+ stats.managedLedgerStats.storageLogicalSize, cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_backlog_size", stats.managedLedgerStats.backlogSize, cluster,
+ namespace);
+ writeMetric(allMetrics, "pulsar_storage_offloaded_size",
+ stats.managedLedgerStats.offloadedStorageUsed, cluster, namespace);
+
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_storage_write_rate", stats.managedLedgerStats.storageWriteRate,
+ cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_storage_read_rate", stats.managedLedgerStats.storageReadRate,
+ cluster, namespace);
+
+ writeMetric(allMetrics, "pulsar_subscription_delayed", stats.msgDelayed, cluster, namespace);
+
+ writePulsarMsgBacklog(allMetrics, stats.msgBacklog, cluster, namespace);
- private static void printNamespaceStats(SimpleTextOutputStream stream, String cluster, String namespace,
- AggregatedNamespaceStats stats) {
- metric(stream, cluster, namespace, "pulsar_topics_count", stats.topicsCount);
- metric(stream, cluster, namespace, "pulsar_subscriptions_count", stats.subscriptionsCount);
- metric(stream, cluster, namespace, "pulsar_producers_count", stats.producersCount);
- metric(stream, cluster, namespace, "pulsar_consumers_count", stats.consumersCount);
+ stats.managedLedgerStats.storageWriteLatencyBuckets.refresh();
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_0_5",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[0], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_1",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[1], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_5",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[2], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_10",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[3], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_20",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[4], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_50",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[5], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_100",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[6], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_200",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[7], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_1000",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[8], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_overflow",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[9], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_count",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getCount(), cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_sum",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getSum(), cluster, namespace);
- metric(stream, cluster, namespace, "pulsar_rate_in", stats.rateIn);
- metric(stream, cluster, namespace, "pulsar_rate_out", stats.rateOut);
- metric(stream, cluster, namespace, "pulsar_throughput_in", stats.throughputIn);
- metric(stream, cluster, namespace, "pulsar_throughput_out", stats.throughputOut);
- metric(stream, cluster, namespace, "pulsar_consumer_msg_ack_rate", stats.messageAckRate);
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.refresh();
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_0_5",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[0], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_1",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[1], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_5",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[2], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_10",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[3], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_20",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[4], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_50",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[5], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_100",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[6], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_200",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[7], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_1000",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[8], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_overflow",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[9], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_count",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getCount(), cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_sum",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getSum(), cluster, namespace);
- metric(stream, cluster, namespace, "pulsar_in_bytes_total", stats.bytesInCounter);
- metric(stream, cluster, namespace, "pulsar_in_messages_total", stats.msgInCounter);
- metric(stream, cluster, namespace, "pulsar_out_bytes_total", stats.bytesOutCounter);
- metric(stream, cluster, namespace, "pulsar_out_messages_total", stats.msgOutCounter);
+ stats.managedLedgerStats.entrySizeBuckets.refresh();
+ writeMetric(allMetrics, "pulsar_entry_size_le_128", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[0],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_512", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[1],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_1_kb", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[2],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_2_kb", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[3],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_4_kb", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[4],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_16_kb", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[5],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_100_kb",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[6], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_1_mb", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[7],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_overflow",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[8], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_count",
+ stats.managedLedgerStats.entrySizeBuckets.getCount(), cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_sum",
+ stats.managedLedgerStats.entrySizeBuckets.getSum(), cluster, namespace);
+
+ writeReplicationStat(allMetrics, "pulsar_replication_rate_in", stats,
+ replStats -> replStats.msgRateIn, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_rate_out", stats,
+ replStats -> replStats.msgRateOut, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_throughput_in", stats,
+ replStats -> replStats.msgThroughputIn, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_throughput_out", stats,
+ replStats -> replStats.msgThroughputOut, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_backlog", stats,
+ replStats -> replStats.replicationBacklog, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_connected_count", stats,
+ replStats -> replStats.connectedCount, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_rate_expired", stats,
+ replStats -> replStats.msgRateExpired, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_delay_in_seconds", stats,
+ replStats -> replStats.replicationDelayInSeconds, cluster, namespace);
+ }
- metric(stream, cluster, namespace, "pulsar_storage_size", stats.managedLedgerStats.storageSize);
- metric(stream, cluster, namespace, "pulsar_storage_logical_size", stats.managedLedgerStats.storageLogicalSize);
- metric(stream, cluster, namespace, "pulsar_storage_backlog_size", stats.managedLedgerStats.backlogSize);
- metric(stream, cluster, namespace, "pulsar_storage_offloaded_size",
- stats.managedLedgerStats.offloadedStorageUsed);
+ private static void writeMetricWithBrokerDefault(Map<String, ByteBuf> allMetrics, String metricName, Number value,
+ String cluster, String namespace) {
+ ByteBuf buffer = writeGaugeTypeWithBrokerDefault(allMetrics, metricName, cluster);
+ writeSample(buffer, metricName, value, "cluster", cluster, "namespace", namespace);
+ }
- metric(stream, cluster, namespace, "pulsar_storage_write_rate", stats.managedLedgerStats.storageWriteRate);
- metric(stream, cluster, namespace, "pulsar_storage_read_rate", stats.managedLedgerStats.storageReadRate);
+ private static void writePulsarMsgBacklog(Map<String, ByteBuf> allMetrics, Number value,
+ String cluster, String namespace) {
+ ByteBuf buffer = writeGaugeTypeWithBrokerDefault(allMetrics, "pulsar_msg_backlog", cluster);
+ writeSample(buffer, "pulsar_msg_backlog", value, "cluster", cluster, "namespace", namespace, "remote_cluster",
+ "local");
+ }
- metric(stream, cluster, namespace, "pulsar_subscription_delayed", stats.msgDelayed);
+ private static void writeMetric(Map<String, ByteBuf> allMetrics, String metricName, Number value, String cluster,
+ String namespace) {
+ ByteBuf buffer = writeGaugeType(allMetrics, metricName);
+ writeSample(buffer, metricName, value, "cluster", cluster, "namespace", namespace);
+ }
- metricWithRemoteCluster(stream, cluster, namespace, "pulsar_msg_backlog", "local", stats.msgBacklog);
+ private static void writeReplicationStat(Map<String, ByteBuf> allMetrics, String metricName,
+ AggregatedNamespaceStats namespaceStats,
+ Function<AggregatedReplicationStats, Number> replStatsFunction,
+ String cluster, String namespace) {
+ if (!namespaceStats.replicationStats.isEmpty()) {
+ ByteBuf buffer = writeGaugeType(allMetrics, metricName);
+ namespaceStats.replicationStats.forEach((remoteCluster, replStats) ->
+ writeSample(buffer, metricName, replStatsFunction.apply(replStats),
+ "cluster", cluster,
+ "namespace", namespace,
+ "remote_cluster", remoteCluster)
+ );
+ }
+ }
- stats.managedLedgerStats.storageWriteLatencyBuckets.refresh();
- long[] latencyBuckets = stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets();
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_0_5", latencyBuckets[0]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_1", latencyBuckets[1]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_5", latencyBuckets[2]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_10", latencyBuckets[3]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_20", latencyBuckets[4]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_50", latencyBuckets[5]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_100", latencyBuckets[6]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_200", latencyBuckets[7]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_1000", latencyBuckets[8]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_overflow", latencyBuckets[9]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_count",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getCount());
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_sum",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getSum());
+ static ByteBuf writeGaugeType(Map<String, ByteBuf> allMetrics, String metricName) {
+ if (!allMetrics.containsKey(metricName)) {
+ ByteBuf buffer = UnpooledByteBufAllocator.DEFAULT.compositeDirectBuffer(MAX_COMPONENTS);
+ write(buffer, "# TYPE ");
+ write(buffer, metricName);
+ write(buffer, " gauge\n");
+ allMetrics.put(metricName, buffer);
+ }
+ return allMetrics.get(metricName);
+ }
- stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.refresh();
- long[] ledgerWritelatencyBuckets = stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets();
- metric(stream, cluster, namespace, "pulsar_storage_ledger_write_latency_le_0_5", ledgerWritelatencyBuckets[0]);
- metric(stream, cluster, namespace, "pulsar_storage_ledger_write_latency_le_1", ledgerWritelatencyBuckets[1]);
- metric(stream, cluster, namespace, "pulsar_storage_ledger_write_latency_le_5", ledgerWritelatencyBuckets[2]);
- metric(stream, cluster, namespace, "pulsar_storage_ledger_write_latency_le_10", ledgerWritelatencyBuckets[3]);
- metric(stream, cluster, namespace, "pulsar_storage_ledger_write_latency_le_20", ledgerWritelatencyBuckets[4]);
- metric(stream, cluster, namespace, "pulsar_storage_ledger_write_latency_le_50", ledgerWritelatencyBuckets[5]);
- metric(stream, cluster, namespace, "pulsar_storage_ledger_write_latency_le_100", ledgerWritelatencyBuckets[6]);
- metric(stream, cluster, namespace, "pulsar_storage_ledger_write_latency_le_200", ledgerWritelatencyBuckets[7]);
- metric(stream, cluster, namespace, "pulsar_storage_ledger_write_latency_le_1000", ledgerWritelatencyBuckets[8]);
- metric(stream, cluster, namespace, "pulsar_storage_ledger_write_latency_overflow",
- ledgerWritelatencyBuckets[9]);
- metric(stream, cluster, namespace, "pulsar_storage_ledger_write_latency_count",
- stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getCount());
- metric(stream, cluster, namespace, "pulsar_storage_ledger_write_latency_sum",
- stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getSum());
+ static ByteBuf writeGaugeTypeWithBrokerDefault(Map<String, ByteBuf> allMetrics, String metricName, String cluster) {
+ if (!allMetrics.containsKey(metricName)) {
+ ByteBuf buffer = writeGaugeType(allMetrics, metricName);
+ writeSample(buffer, metricName, 0, "cluster", cluster);
+ }
+ return allMetrics.get(metricName);
+ }
- stats.managedLedgerStats.entrySizeBuckets.refresh();
- long[] entrySizeBuckets = stats.managedLedgerStats.entrySizeBuckets.getBuckets();
- metric(stream, cluster, namespace, "pulsar_entry_size_le_128", entrySizeBuckets[0]);
- metric(stream, cluster, namespace, "pulsar_entry_size_le_512", entrySizeBuckets[1]);
- metric(stream, cluster, namespace, "pulsar_entry_size_le_1_kb", entrySizeBuckets[2]);
- metric(stream, cluster, namespace, "pulsar_entry_size_le_2_kb", entrySizeBuckets[3]);
- metric(stream, cluster, namespace, "pulsar_entry_size_le_4_kb", entrySizeBuckets[4]);
- metric(stream, cluster, namespace, "pulsar_entry_size_le_16_kb", entrySizeBuckets[5]);
- metric(stream, cluster, namespace, "pulsar_entry_size_le_100_kb", entrySizeBuckets[6]);
- metric(stream, cluster, namespace, "pulsar_entry_size_le_1_mb", entrySizeBuckets[7]);
- metric(stream, cluster, namespace, "pulsar_entry_size_le_overflow", entrySizeBuckets[8]);
- metric(stream, cluster, namespace, "pulsar_entry_size_count",
- stats.managedLedgerStats.entrySizeBuckets.getCount());
- metric(stream, cluster, namespace, "pulsar_entry_size_sum",
- stats.managedLedgerStats.entrySizeBuckets.getSum());
-
- if (!stats.replicationStats.isEmpty()) {
- stats.replicationStats.forEach((remoteCluster, replStats) -> {
- metricWithRemoteCluster(stream, cluster, namespace, "pulsar_replication_rate_in", remoteCluster,
- replStats.msgRateIn);
- metricWithRemoteCluster(stream, cluster, namespace, "pulsar_replication_rate_out", remoteCluster,
- replStats.msgRateOut);
- metricWithRemoteCluster(stream, cluster, namespace, "pulsar_replication_throughput_in", remoteCluster,
- replStats.msgThroughputIn);
- metricWithRemoteCluster(stream, cluster, namespace, "pulsar_replication_throughput_out", remoteCluster,
- replStats.msgThroughputOut);
- metricWithRemoteCluster(stream, cluster, namespace, "pulsar_replication_backlog", remoteCluster,
- replStats.replicationBacklog);
- metricWithRemoteCluster(stream, cluster, namespace, "pulsar_replication_connected_count", remoteCluster,
- replStats.connectedCount);
- metricWithRemoteCluster(stream, cluster, namespace, "pulsar_replication_rate_expired", remoteCluster,
- replStats.msgRateExpired);
- metricWithRemoteCluster(stream, cluster, namespace, "pulsar_replication_delay_in_seconds",
- remoteCluster, replStats.replicationDelayInSeconds);
- });
+ static void writeSample(ByteBuf buffer, String metricName, Number value, String... labelsAndValuesArray) {
+ write(buffer, metricName);
+ write(buffer, '{');
+ for (int i = 0; i < labelsAndValuesArray.length; i += 2) {
+ write(buffer, labelsAndValuesArray[i]);
+ write(buffer, "=\"");
+ write(buffer, labelsAndValuesArray[i + 1]);
+ write(buffer, '\"');
+ if (labelsAndValuesArray.length != i + 2) {
Review Comment:
`labelsAndValuesArray.length != i + 2` --> `i + 2 != labelsAndValuesArray.length`
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/NamespaceStatsAggregator.java:
##########
@@ -296,161 +306,264 @@ private static void getTopicStats(Topic topic, TopicStats stats, boolean include
});
}
- private static void printDefaultBrokerStats(SimpleTextOutputStream stream, String cluster) {
- // Print metrics with 0 values. This is necessary to have the available brokers being
- // reported in the brokers dashboard even if they don't have any topic or traffic
- metric(stream, cluster, "pulsar_topics_count", 0);
- metric(stream, cluster, "pulsar_subscriptions_count", 0);
- metric(stream, cluster, "pulsar_producers_count", 0);
- metric(stream, cluster, "pulsar_consumers_count", 0);
- metric(stream, cluster, "pulsar_rate_in", 0);
- metric(stream, cluster, "pulsar_rate_out", 0);
- metric(stream, cluster, "pulsar_throughput_in", 0);
- metric(stream, cluster, "pulsar_throughput_out", 0);
- metric(stream, cluster, "pulsar_storage_size", 0);
- metric(stream, cluster, "pulsar_storage_logical_size", 0);
- metric(stream, cluster, "pulsar_storage_write_rate", 0);
- metric(stream, cluster, "pulsar_storage_read_rate", 0);
- metric(stream, cluster, "pulsar_msg_backlog", 0);
+ private static void printTopicsCountStats(Map<String, ByteBuf> allMetrics, Map<String, Long> namespaceTopicsCount,
+ String cluster) {
+ namespaceTopicsCount.forEach((ns, topicCount) ->
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_topics_count", topicCount, cluster, ns)
+ );
}
- private static void printTopicsCountStats(SimpleTextOutputStream stream, String cluster, String namespace,
- LongAdder topicsCount) {
- metric(stream, cluster, namespace, "pulsar_topics_count", topicsCount.sum());
- }
+ private static void printNamespaceStats(Map<String, ByteBuf> allMetrics, AggregatedNamespaceStats stats,
+ String cluster, String namespace) {
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_topics_count", stats.topicsCount, cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_subscriptions_count", stats.subscriptionsCount, cluster,
+ namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_producers_count", stats.producersCount, cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_consumers_count", stats.consumersCount, cluster, namespace);
+
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_rate_in", stats.rateIn, cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_rate_out", stats.rateOut, cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_throughput_in", stats.throughputIn, cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_throughput_out", stats.throughputOut, cluster, namespace);
+ writeMetric(allMetrics, "pulsar_consumer_msg_ack_rate", stats.messageAckRate, cluster, namespace);
+
+ writeMetric(allMetrics, "pulsar_in_bytes_total", stats.bytesInCounter, cluster, namespace);
+ writeMetric(allMetrics, "pulsar_in_messages_total", stats.msgInCounter, cluster, namespace);
+ writeMetric(allMetrics, "pulsar_out_bytes_total", stats.bytesOutCounter, cluster, namespace);
+ writeMetric(allMetrics, "pulsar_out_messages_total", stats.msgOutCounter, cluster, namespace);
+
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_storage_size", stats.managedLedgerStats.storageSize, cluster,
+ namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_storage_logical_size",
+ stats.managedLedgerStats.storageLogicalSize, cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_backlog_size", stats.managedLedgerStats.backlogSize, cluster,
+ namespace);
+ writeMetric(allMetrics, "pulsar_storage_offloaded_size",
+ stats.managedLedgerStats.offloadedStorageUsed, cluster, namespace);
+
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_storage_write_rate", stats.managedLedgerStats.storageWriteRate,
+ cluster, namespace);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_storage_read_rate", stats.managedLedgerStats.storageReadRate,
+ cluster, namespace);
+
+ writeMetric(allMetrics, "pulsar_subscription_delayed", stats.msgDelayed, cluster, namespace);
+
+ writePulsarMsgBacklog(allMetrics, stats.msgBacklog, cluster, namespace);
- private static void printNamespaceStats(SimpleTextOutputStream stream, String cluster, String namespace,
- AggregatedNamespaceStats stats) {
- metric(stream, cluster, namespace, "pulsar_topics_count", stats.topicsCount);
- metric(stream, cluster, namespace, "pulsar_subscriptions_count", stats.subscriptionsCount);
- metric(stream, cluster, namespace, "pulsar_producers_count", stats.producersCount);
- metric(stream, cluster, namespace, "pulsar_consumers_count", stats.consumersCount);
+ stats.managedLedgerStats.storageWriteLatencyBuckets.refresh();
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_0_5",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[0], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_1",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[1], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_5",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[2], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_10",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[3], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_20",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[4], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_50",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[5], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_100",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[6], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_200",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[7], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_1000",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[8], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_overflow",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[9], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_count",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getCount(), cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_sum",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getSum(), cluster, namespace);
- metric(stream, cluster, namespace, "pulsar_rate_in", stats.rateIn);
- metric(stream, cluster, namespace, "pulsar_rate_out", stats.rateOut);
- metric(stream, cluster, namespace, "pulsar_throughput_in", stats.throughputIn);
- metric(stream, cluster, namespace, "pulsar_throughput_out", stats.throughputOut);
- metric(stream, cluster, namespace, "pulsar_consumer_msg_ack_rate", stats.messageAckRate);
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.refresh();
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_0_5",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[0], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_1",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[1], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_5",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[2], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_10",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[3], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_20",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[4], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_50",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[5], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_100",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[6], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_200",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[7], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_1000",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[8], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_overflow",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[9], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_count",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getCount(), cluster, namespace);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_sum",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getSum(), cluster, namespace);
- metric(stream, cluster, namespace, "pulsar_in_bytes_total", stats.bytesInCounter);
- metric(stream, cluster, namespace, "pulsar_in_messages_total", stats.msgInCounter);
- metric(stream, cluster, namespace, "pulsar_out_bytes_total", stats.bytesOutCounter);
- metric(stream, cluster, namespace, "pulsar_out_messages_total", stats.msgOutCounter);
+ stats.managedLedgerStats.entrySizeBuckets.refresh();
+ writeMetric(allMetrics, "pulsar_entry_size_le_128", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[0],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_512", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[1],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_1_kb", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[2],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_2_kb", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[3],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_4_kb", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[4],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_16_kb", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[5],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_100_kb",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[6], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_1_mb", stats.managedLedgerStats.entrySizeBuckets.getBuckets()[7],
+ cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_le_overflow",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[8], cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_count",
+ stats.managedLedgerStats.entrySizeBuckets.getCount(), cluster, namespace);
+ writeMetric(allMetrics, "pulsar_entry_size_sum",
+ stats.managedLedgerStats.entrySizeBuckets.getSum(), cluster, namespace);
+
+ writeReplicationStat(allMetrics, "pulsar_replication_rate_in", stats,
+ replStats -> replStats.msgRateIn, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_rate_out", stats,
+ replStats -> replStats.msgRateOut, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_throughput_in", stats,
+ replStats -> replStats.msgThroughputIn, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_throughput_out", stats,
+ replStats -> replStats.msgThroughputOut, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_backlog", stats,
+ replStats -> replStats.replicationBacklog, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_connected_count", stats,
+ replStats -> replStats.connectedCount, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_rate_expired", stats,
+ replStats -> replStats.msgRateExpired, cluster, namespace);
+ writeReplicationStat(allMetrics, "pulsar_replication_delay_in_seconds", stats,
+ replStats -> replStats.replicationDelayInSeconds, cluster, namespace);
+ }
- metric(stream, cluster, namespace, "pulsar_storage_size", stats.managedLedgerStats.storageSize);
- metric(stream, cluster, namespace, "pulsar_storage_logical_size", stats.managedLedgerStats.storageLogicalSize);
- metric(stream, cluster, namespace, "pulsar_storage_backlog_size", stats.managedLedgerStats.backlogSize);
- metric(stream, cluster, namespace, "pulsar_storage_offloaded_size",
- stats.managedLedgerStats.offloadedStorageUsed);
+ private static void writeMetricWithBrokerDefault(Map<String, ByteBuf> allMetrics, String metricName, Number value,
+ String cluster, String namespace) {
+ ByteBuf buffer = writeGaugeTypeWithBrokerDefault(allMetrics, metricName, cluster);
+ writeSample(buffer, metricName, value, "cluster", cluster, "namespace", namespace);
+ }
- metric(stream, cluster, namespace, "pulsar_storage_write_rate", stats.managedLedgerStats.storageWriteRate);
- metric(stream, cluster, namespace, "pulsar_storage_read_rate", stats.managedLedgerStats.storageReadRate);
+ private static void writePulsarMsgBacklog(Map<String, ByteBuf> allMetrics, Number value,
+ String cluster, String namespace) {
+ ByteBuf buffer = writeGaugeTypeWithBrokerDefault(allMetrics, "pulsar_msg_backlog", cluster);
+ writeSample(buffer, "pulsar_msg_backlog", value, "cluster", cluster, "namespace", namespace, "remote_cluster",
+ "local");
+ }
- metric(stream, cluster, namespace, "pulsar_subscription_delayed", stats.msgDelayed);
+ private static void writeMetric(Map<String, ByteBuf> allMetrics, String metricName, Number value, String cluster,
+ String namespace) {
+ ByteBuf buffer = writeGaugeType(allMetrics, metricName);
+ writeSample(buffer, metricName, value, "cluster", cluster, "namespace", namespace);
+ }
- metricWithRemoteCluster(stream, cluster, namespace, "pulsar_msg_backlog", "local", stats.msgBacklog);
+ private static void writeReplicationStat(Map<String, ByteBuf> allMetrics, String metricName,
+ AggregatedNamespaceStats namespaceStats,
+ Function<AggregatedReplicationStats, Number> replStatsFunction,
+ String cluster, String namespace) {
+ if (!namespaceStats.replicationStats.isEmpty()) {
+ ByteBuf buffer = writeGaugeType(allMetrics, metricName);
+ namespaceStats.replicationStats.forEach((remoteCluster, replStats) ->
+ writeSample(buffer, metricName, replStatsFunction.apply(replStats),
+ "cluster", cluster,
+ "namespace", namespace,
+ "remote_cluster", remoteCluster)
+ );
+ }
+ }
- stats.managedLedgerStats.storageWriteLatencyBuckets.refresh();
- long[] latencyBuckets = stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets();
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_0_5", latencyBuckets[0]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_1", latencyBuckets[1]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_5", latencyBuckets[2]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_10", latencyBuckets[3]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_20", latencyBuckets[4]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_50", latencyBuckets[5]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_100", latencyBuckets[6]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_200", latencyBuckets[7]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_le_1000", latencyBuckets[8]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_overflow", latencyBuckets[9]);
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_count",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getCount());
- metric(stream, cluster, namespace, "pulsar_storage_write_latency_sum",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getSum());
+ static ByteBuf writeGaugeType(Map<String, ByteBuf> allMetrics, String metricName) {
+ if (!allMetrics.containsKey(metricName)) {
+ ByteBuf buffer = UnpooledByteBufAllocator.DEFAULT.compositeDirectBuffer(MAX_COMPONENTS);
+ write(buffer, "# TYPE ");
+ write(buffer, metricName);
+ write(buffer, " gauge\n");
+ allMetrics.put(metricName, buffer);
+ }
+ return allMetrics.get(metricName);
+ }
- stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.refresh();
- long[] ledgerWritelatencyBuckets = stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets();
- metric(stream, cluster, namespace, "pulsar_storage_ledger_write_latency_le_0_5", ledgerWritelatencyBuckets[0]);
- metric(stream, cluster, namespace, "pulsar_storage_ledger_write_latency_le_1", ledgerWritelatencyBuckets[1]);
- metric(stream, cluster, namespace, "pulsar_storage_ledger_write_latency_le_5", ledgerWritelatencyBuckets[2]);
- metric(stream, cluster, namespace, "pulsar_storage_ledger_write_latency_le_10", ledgerWritelatencyBuckets[3]);
- metric(stream, cluster, namespace, "pulsar_storage_ledger_write_latency_le_20", ledgerWritelatencyBuckets[4]);
- metric(stream, cluster, namespace, "pulsar_storage_ledger_write_latency_le_50", ledgerWritelatencyBuckets[5]);
- metric(stream, cluster, namespace, "pulsar_storage_ledger_write_latency_le_100", ledgerWritelatencyBuckets[6]);
- metric(stream, cluster, namespace, "pulsar_storage_ledger_write_latency_le_200", ledgerWritelatencyBuckets[7]);
- metric(stream, cluster, namespace, "pulsar_storage_ledger_write_latency_le_1000", ledgerWritelatencyBuckets[8]);
- metric(stream, cluster, namespace, "pulsar_storage_ledger_write_latency_overflow",
- ledgerWritelatencyBuckets[9]);
- metric(stream, cluster, namespace, "pulsar_storage_ledger_write_latency_count",
- stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getCount());
- metric(stream, cluster, namespace, "pulsar_storage_ledger_write_latency_sum",
- stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getSum());
+ static ByteBuf writeGaugeTypeWithBrokerDefault(Map<String, ByteBuf> allMetrics, String metricName, String cluster) {
+ if (!allMetrics.containsKey(metricName)) {
+ ByteBuf buffer = writeGaugeType(allMetrics, metricName);
+ writeSample(buffer, metricName, 0, "cluster", cluster);
+ }
+ return allMetrics.get(metricName);
+ }
- stats.managedLedgerStats.entrySizeBuckets.refresh();
- long[] entrySizeBuckets = stats.managedLedgerStats.entrySizeBuckets.getBuckets();
- metric(stream, cluster, namespace, "pulsar_entry_size_le_128", entrySizeBuckets[0]);
- metric(stream, cluster, namespace, "pulsar_entry_size_le_512", entrySizeBuckets[1]);
- metric(stream, cluster, namespace, "pulsar_entry_size_le_1_kb", entrySizeBuckets[2]);
- metric(stream, cluster, namespace, "pulsar_entry_size_le_2_kb", entrySizeBuckets[3]);
- metric(stream, cluster, namespace, "pulsar_entry_size_le_4_kb", entrySizeBuckets[4]);
- metric(stream, cluster, namespace, "pulsar_entry_size_le_16_kb", entrySizeBuckets[5]);
- metric(stream, cluster, namespace, "pulsar_entry_size_le_100_kb", entrySizeBuckets[6]);
- metric(stream, cluster, namespace, "pulsar_entry_size_le_1_mb", entrySizeBuckets[7]);
- metric(stream, cluster, namespace, "pulsar_entry_size_le_overflow", entrySizeBuckets[8]);
- metric(stream, cluster, namespace, "pulsar_entry_size_count",
- stats.managedLedgerStats.entrySizeBuckets.getCount());
- metric(stream, cluster, namespace, "pulsar_entry_size_sum",
- stats.managedLedgerStats.entrySizeBuckets.getSum());
-
- if (!stats.replicationStats.isEmpty()) {
- stats.replicationStats.forEach((remoteCluster, replStats) -> {
- metricWithRemoteCluster(stream, cluster, namespace, "pulsar_replication_rate_in", remoteCluster,
- replStats.msgRateIn);
- metricWithRemoteCluster(stream, cluster, namespace, "pulsar_replication_rate_out", remoteCluster,
- replStats.msgRateOut);
- metricWithRemoteCluster(stream, cluster, namespace, "pulsar_replication_throughput_in", remoteCluster,
- replStats.msgThroughputIn);
- metricWithRemoteCluster(stream, cluster, namespace, "pulsar_replication_throughput_out", remoteCluster,
- replStats.msgThroughputOut);
- metricWithRemoteCluster(stream, cluster, namespace, "pulsar_replication_backlog", remoteCluster,
- replStats.replicationBacklog);
- metricWithRemoteCluster(stream, cluster, namespace, "pulsar_replication_connected_count", remoteCluster,
- replStats.connectedCount);
- metricWithRemoteCluster(stream, cluster, namespace, "pulsar_replication_rate_expired", remoteCluster,
- replStats.msgRateExpired);
- metricWithRemoteCluster(stream, cluster, namespace, "pulsar_replication_delay_in_seconds",
- remoteCluster, replStats.replicationDelayInSeconds);
- });
+ static void writeSample(ByteBuf buffer, String metricName, Number value, String... labelsAndValuesArray) {
+ write(buffer, metricName);
+ write(buffer, '{');
+ for (int i = 0; i < labelsAndValuesArray.length; i += 2) {
+ write(buffer, labelsAndValuesArray[i]);
+ write(buffer, "=\"");
+ write(buffer, labelsAndValuesArray[i + 1]);
+ write(buffer, '\"');
+ if (labelsAndValuesArray.length != i + 2) {
+ write(buffer, ',');
+ }
+ }
+ write(buffer, "\"} ");
+ write(buffer, value);
+ write(buffer, ' ');
+ write(buffer, System.currentTimeMillis());
+ write(buffer, '\n');
+ }
+
+ private static void write(ByteBuf buffer, String s) {
Review Comment:
As wrote above, all those functions can be deleted if we use `SimpleTextOutputFormat` instead of `ByteBuf`, as cardinality is not at names of metrics.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/TopicStats.java:
##########
@@ -102,380 +104,401 @@ public void reset() {
compactionLatencyBuckets.reset();
}
- static void resetTypes() {
- metricWithTypeDefinition.clear();
- }
-
- static void printTopicStats(SimpleTextOutputStream stream, String cluster, String namespace, String topic,
- TopicStats stats, Optional<CompactorMXBean> compactorMXBean,
- boolean splitTopicAndPartitionIndexLabel) {
- metric(stream, cluster, namespace, topic, "pulsar_subscriptions_count", stats.subscriptionsCount,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_producers_count", stats.producersCount,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_consumers_count", stats.consumersCount,
- splitTopicAndPartitionIndexLabel);
-
- metric(stream, cluster, namespace, topic, "pulsar_rate_in", stats.rateIn,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_rate_out", stats.rateOut,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_throughput_in", stats.throughputIn,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_throughput_out", stats.throughputOut,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_average_msg_size", stats.averageMsgSize,
- splitTopicAndPartitionIndexLabel);
-
- metric(stream, cluster, namespace, topic, "pulsar_storage_size", stats.managedLedgerStats.storageSize,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_logical_size",
- stats.managedLedgerStats.storageLogicalSize, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_msg_backlog", stats.msgBacklog,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_backlog_size",
- stats.managedLedgerStats.backlogSize, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_publish_rate_limit_times", stats.publishRateLimitedTimes,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_offloaded_size", stats.managedLedgerStats
- .offloadedStorageUsed, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_backlog_quota_limit", stats.backlogQuotaLimit,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_backlog_quota_limit_time",
- stats.backlogQuotaLimitTime, splitTopicAndPartitionIndexLabel);
-
- long[] latencyBuckets = stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets();
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_le_0_5", latencyBuckets[0],
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_le_1", latencyBuckets[1],
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_le_5", latencyBuckets[2],
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_le_10", latencyBuckets[3],
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_le_20", latencyBuckets[4],
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_le_50", latencyBuckets[5],
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_le_100", latencyBuckets[6],
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_le_200", latencyBuckets[7],
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_le_1000", latencyBuckets[8],
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_overflow", latencyBuckets[9],
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_count",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getCount(), splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_sum",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getSum(), splitTopicAndPartitionIndexLabel);
-
- long[] ledgerWriteLatencyBuckets = stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets();
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_0_5",
- ledgerWriteLatencyBuckets[0], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_1",
- ledgerWriteLatencyBuckets[1], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_5",
- ledgerWriteLatencyBuckets[2], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_10",
- ledgerWriteLatencyBuckets[3], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_20",
- ledgerWriteLatencyBuckets[4], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_50",
- ledgerWriteLatencyBuckets[5], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_100",
- ledgerWriteLatencyBuckets[6], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_200",
- ledgerWriteLatencyBuckets[7], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_1000",
- ledgerWriteLatencyBuckets[8], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_overflow",
- ledgerWriteLatencyBuckets[9], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_count",
+ public static void printTopicStats(Map<String, ByteBuf> allMetrics, TopicStats stats,
+ Optional<CompactorMXBean> compactorMXBean, String cluster, String namespace,
+ String name, boolean splitTopicAndPartitionIndexLabel) {
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_subscriptions_count", stats.subscriptionsCount,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_producers_count", stats.producersCount,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_consumers_count", stats.consumersCount,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_rate_in", stats.rateIn,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_rate_out", stats.rateOut,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_throughput_in", stats.throughputIn,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_throughput_out", stats.throughputOut,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_average_msg_size", stats.averageMsgSize,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_storage_size", stats.managedLedgerStats.storageSize,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_storage_logical_size",
+ stats.managedLedgerStats.storageLogicalSize, cluster, namespace, name,
+ splitTopicAndPartitionIndexLabel);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_msg_backlog", stats.msgBacklog,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_backlog_size", stats.managedLedgerStats.backlogSize,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_publish_rate_limit_times", stats.publishRateLimitedTimes,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_offloaded_size", stats.managedLedgerStats
+ .offloadedStorageUsed, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_backlog_quota_limit", stats.backlogQuotaLimit,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_backlog_quota_limit_time", stats.backlogQuotaLimitTime,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_0_5",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[0],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_1",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[1],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_5",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[2],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_10",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[3],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_20",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[4],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_50",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[5],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_100",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[6],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_200",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[7],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_1000",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[8],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_overflow",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[9],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_count",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getCount(),
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_sum",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getSum(), cluster, namespace, name,
+ splitTopicAndPartitionIndexLabel);
+
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_0_5",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[0],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_1",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[1],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_5",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[2],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_10",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[3],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_20",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[4],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_50",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[5],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_100",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[6],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_200",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[7],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_1000",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[8],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_overflow",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[9],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_count",
stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getCount(),
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_sum",
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_sum",
stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getSum(),
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+ writeMetric(allMetrics, "pulsar_entry_size_le_128",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[0],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_entry_size_le_512",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[1],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_entry_size_le_1_kb",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[2],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_entry_size_le_2_kb",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[3],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_entry_size_le_4_kb",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[4],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_entry_size_le_16_kb",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[5],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_entry_size_le_100_kb",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[6],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_entry_size_le_1_mb",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[7],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_entry_size_le_overflow",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[8],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_entry_size_count", stats.managedLedgerStats.entrySizeBuckets.getCount(),
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_entry_size_sum", stats.managedLedgerStats.entrySizeBuckets.getSum(),
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+ writeProducerStat(allMetrics, "pulsar_producer_msg_rate_in", stats,
+ p -> p.msgRateIn, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeProducerStat(allMetrics, "pulsar_producer_msg_throughput_in", stats,
+ p -> p.msgThroughputIn, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeProducerStat(allMetrics, "pulsar_producer_msg_average_Size", stats,
+ p -> p.averageMsgSize, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_back_log", stats, s -> s.msgBacklog,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_back_log_no_delayed",
+ stats, s -> s.msgBacklogNoDelayed, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_delayed",
+ stats, s -> s.msgDelayed, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_msg_rate_redeliver",
+ stats, s -> s.msgRateRedeliver, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_unacked_messages",
+ stats, s -> s.unackedMessages, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_blocked_on_unacked_messages",
+ stats, s -> s.blockedSubscriptionOnUnackedMsgs ? 1 : 0, cluster, namespace, name,
+ splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_msg_rate_out",
+ stats, s -> s.msgRateOut, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_msg_ack_rate",
+ stats, s -> s.messageAckRate, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_msg_throughput_out",
+ stats, s -> s.msgThroughputOut, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_out_bytes_total",
+ stats, s -> s.bytesOutCounter, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_out_messages_total",
+ stats, s -> s.msgOutCounter, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_last_expire_timestamp",
+ stats, s -> s.lastExpireTimestamp, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_last_acked_timestamp",
+ stats, s -> s.lastAckedTimestamp, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_last_consumed_flow_timestamp",
+ stats, s -> s.lastConsumedFlowTimestamp, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_last_consumed_timestamp",
+ stats, s -> s.lastConsumedTimestamp, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_last_mark_delete_advanced_timestamp",
+ stats, s -> s.lastMarkDeleteAdvancedTimestamp, cluster, namespace, name,
+ splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_msg_rate_expired",
+ stats, s -> s.msgRateExpired, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_total_msg_expired",
+ stats, s -> s.totalMsgExpired, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_msg_drop_rate",
+ stats, s -> s.msgDropRate, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_consumers_count",
+ stats, s -> s.consumersCount, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+ writeConsumerStat(allMetrics, "pulsar_consumer_msg_rate_redeliver", stats, c -> c.msgRateRedeliver,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeConsumerStat(allMetrics, "pulsar_consumer_unacked_messages", stats, c -> c.unackedMessages,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeConsumerStat(allMetrics, "pulsar_consumer_blocked_on_unacked_messages",
+ stats, c -> c.blockedSubscriptionOnUnackedMsgs ? 1 : 0,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeConsumerStat(allMetrics, "pulsar_consumer_msg_rate_out", stats, c -> c.msgRateOut,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+ writeConsumerStat(allMetrics, "pulsar_consumer_msg_ack_rate", stats, c -> c.msgAckRate,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+ writeConsumerStat(allMetrics, "pulsar_consumer_msg_throughput_out", stats, c -> c.msgThroughputOut,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeConsumerStat(allMetrics, "pulsar_consumer_available_permits", stats, c -> c.availablePermits,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeConsumerStat(allMetrics, "pulsar_out_bytes_total", stats, c -> c.bytesOutCounter,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeConsumerStat(allMetrics, "pulsar_out_messages_total", stats, c -> c.msgOutCounter,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+ writeReplicationStat(allMetrics, "pulsar_replication_rate_in", stats, r -> r.msgRateIn,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(allMetrics, "pulsar_replication_rate_out", stats, r -> r.msgRateOut,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(allMetrics, "pulsar_replication_throughput_in", stats, r -> r.msgThroughputIn,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(allMetrics, "pulsar_replication_throughput_out", stats, r -> r.msgThroughputOut,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(allMetrics, "pulsar_replication_backlog", stats, r -> r.replicationBacklog,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(allMetrics, "pulsar_replication_connected_count", stats, r -> r.connectedCount,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(allMetrics, "pulsar_replication_rate_expired", stats, r -> r.msgRateExpired,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(allMetrics, "pulsar_replication_delay_in_seconds", stats,
+ r -> r.replicationDelayInSeconds, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+ writeMetric(allMetrics, "pulsar_in_bytes_total", stats.bytesInCounter, cluster, namespace, name,
+ splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_in_messages_total", stats.msgInCounter, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- long[] entrySizeBuckets = stats.managedLedgerStats.entrySizeBuckets.getBuckets();
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_le_128", entrySizeBuckets[0],
+ // Compaction
+
+ writeCompactionStat(allMetrics, "pulsar_compaction_removed_event_count", stats.compactionRemovedEventCount,
+ compactorMXBean, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(allMetrics, "pulsar_compaction_succeed_count", stats.compactionSucceedCount,
+ compactorMXBean, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(allMetrics, "pulsar_compaction_failed_count", stats.compactionFailedCount, compactorMXBean,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(allMetrics, "pulsar_compaction_duration_time_in_mills", stats.compactionDurationTimeInMills,
+ compactorMXBean, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(allMetrics, "pulsar_compaction_read_throughput", stats.compactionReadThroughput,
+ compactorMXBean, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(allMetrics, "pulsar_compaction_write_throughput", stats.compactionWriteThroughput,
+ compactorMXBean, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(allMetrics, "pulsar_compaction_compacted_entries_count",
+ stats.compactionCompactedEntriesCount, compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_le_512", entrySizeBuckets[1],
+ writeCompactionStat(allMetrics, "pulsar_compaction_compacted_entries_size",
+ stats.compactionCompactedEntriesSize, compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_le_1_kb", entrySizeBuckets[2],
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_le_0_5",
+ stats.compactionLatencyBuckets.getBuckets()[0], compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_le_2_kb", entrySizeBuckets[3],
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_le_1",
+ stats.compactionLatencyBuckets.getBuckets()[1], compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_le_4_kb", entrySizeBuckets[4],
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_le_5",
+ stats.compactionLatencyBuckets.getBuckets()[2], compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_le_16_kb", entrySizeBuckets[5],
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_le_10",
+ stats.compactionLatencyBuckets.getBuckets()[3], compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_le_100_kb", entrySizeBuckets[6],
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_le_20",
+ stats.compactionLatencyBuckets.getBuckets()[4], compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_le_1_mb", entrySizeBuckets[7],
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_le_50",
+ stats.compactionLatencyBuckets.getBuckets()[5], compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_le_overflow", entrySizeBuckets[8],
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_le_100",
+ stats.compactionLatencyBuckets.getBuckets()[6], compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_count",
- stats.managedLedgerStats.entrySizeBuckets.getCount(), splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_sum",
- stats.managedLedgerStats.entrySizeBuckets.getSum(), splitTopicAndPartitionIndexLabel);
-
- stats.producerStats.forEach((p, producerStats) -> {
- metric(stream, cluster, namespace, topic, p, producerStats.producerId, "pulsar_producer_msg_rate_in",
- producerStats.msgRateIn, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, p, producerStats.producerId, "pulsar_producer_msg_throughput_in",
- producerStats.msgThroughputIn, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, p, producerStats.producerId, "pulsar_producer_msg_average_Size",
- producerStats.averageMsgSize, splitTopicAndPartitionIndexLabel);
- });
-
- stats.subscriptionStats.forEach((n, subsStats) -> {
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_back_log",
- subsStats.msgBacklog, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_back_log_no_delayed",
- subsStats.msgBacklogNoDelayed, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_delayed",
- subsStats.msgDelayed, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_msg_rate_redeliver",
- subsStats.msgRateRedeliver, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_unacked_messages",
- subsStats.unackedMessages, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_blocked_on_unacked_messages",
- subsStats.blockedSubscriptionOnUnackedMsgs ? 1 : 0, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_msg_rate_out",
- subsStats.msgRateOut, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_msg_ack_rate",
- subsStats.messageAckRate, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_msg_throughput_out",
- subsStats.msgThroughputOut, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_out_bytes_total",
- subsStats.bytesOutCounter, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_out_messages_total",
- subsStats.msgOutCounter, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_last_expire_timestamp",
- subsStats.lastExpireTimestamp, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_last_acked_timestamp",
- subsStats.lastAckedTimestamp, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_last_consumed_flow_timestamp",
- subsStats.lastConsumedFlowTimestamp, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_last_consumed_timestamp",
- subsStats.lastConsumedTimestamp, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_last_mark_delete_advanced_timestamp",
- subsStats.lastMarkDeleteAdvancedTimestamp, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_msg_rate_expired",
- subsStats.msgRateExpired, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_total_msg_expired",
- subsStats.totalMsgExpired, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_msg_drop_rate",
- subsStats.msgDropRate, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_consumers_count",
- subsStats.consumersCount, splitTopicAndPartitionIndexLabel);
- subsStats.consumerStat.forEach((c, consumerStats) -> {
- metric(stream, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_msg_rate_redeliver", consumerStats.msgRateRedeliver,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_unacked_messages", consumerStats.unackedMessages,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_blocked_on_unacked_messages",
- consumerStats.blockedSubscriptionOnUnackedMsgs ? 1 : 0,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_msg_rate_out", consumerStats.msgRateOut,
- splitTopicAndPartitionIndexLabel);
-
- metric(stream, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_msg_ack_rate", consumerStats.msgAckRate,
- splitTopicAndPartitionIndexLabel);
-
- metric(stream, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_msg_throughput_out", consumerStats.msgThroughputOut,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_available_permits", consumerStats.availablePermits,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_out_bytes_total", consumerStats.bytesOutCounter,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_out_messages_total", consumerStats.msgOutCounter,
- splitTopicAndPartitionIndexLabel);
- });
- });
-
- if (!stats.replicationStats.isEmpty()) {
- stats.replicationStats.forEach((remoteCluster, replStats) -> {
- metricWithRemoteCluster(stream, cluster, namespace, topic, "pulsar_replication_rate_in", remoteCluster,
- replStats.msgRateIn, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(stream, cluster, namespace, topic, "pulsar_replication_rate_out", remoteCluster,
- replStats.msgRateOut, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(stream, cluster, namespace, topic, "pulsar_replication_throughput_in",
- remoteCluster, replStats.msgThroughputIn, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(stream, cluster, namespace, topic, "pulsar_replication_throughput_out",
- remoteCluster, replStats.msgThroughputOut, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(stream, cluster, namespace, topic, "pulsar_replication_backlog", remoteCluster,
- replStats.replicationBacklog, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(stream, cluster, namespace, topic, "pulsar_replication_connected_count",
- remoteCluster, replStats.connectedCount, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(stream, cluster, namespace, topic, "pulsar_replication_rate_expired",
- remoteCluster, replStats.msgRateExpired, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(stream, cluster, namespace, topic, "pulsar_replication_delay_in_seconds",
- remoteCluster, replStats.replicationDelayInSeconds, splitTopicAndPartitionIndexLabel);
- });
- }
-
- metric(stream, cluster, namespace, topic, "pulsar_in_bytes_total", stats.bytesInCounter,
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_le_200",
+ stats.compactionLatencyBuckets.getBuckets()[7], compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_in_messages_total", stats.msgInCounter,
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_le_1000",
+ stats.compactionLatencyBuckets.getBuckets()[8], compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
-
- // Compaction
- boolean hasCompaction = compactorMXBean.flatMap(mxBean -> mxBean.getCompactionRecordForTopic(topic))
- .map(__ -> true).orElse(false);
- if (hasCompaction) {
- metric(stream, cluster, namespace, topic, "pulsar_compaction_removed_event_count",
- stats.compactionRemovedEventCount, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_succeed_count",
- stats.compactionSucceedCount, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_failed_count",
- stats.compactionFailedCount, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_duration_time_in_mills",
- stats.compactionDurationTimeInMills, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_read_throughput",
- stats.compactionReadThroughput, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_write_throughput",
- stats.compactionWriteThroughput, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_compacted_entries_count",
- stats.compactionCompactedEntriesCount, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_compacted_entries_size",
- stats.compactionCompactedEntriesSize, splitTopicAndPartitionIndexLabel);
- long[] compactionLatencyBuckets = stats.compactionLatencyBuckets.getBuckets();
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_le_0_5",
- compactionLatencyBuckets[0], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_le_1",
- compactionLatencyBuckets[1], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_le_5",
- compactionLatencyBuckets[2], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_le_10",
- compactionLatencyBuckets[3], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_le_20",
- compactionLatencyBuckets[4], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_le_50",
- compactionLatencyBuckets[5], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_le_100",
- compactionLatencyBuckets[6], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_le_200",
- compactionLatencyBuckets[7], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_le_1000",
- compactionLatencyBuckets[8], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_overflow",
- compactionLatencyBuckets[9], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_sum",
- stats.compactionLatencyBuckets.getSum(), splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_count",
- stats.compactionLatencyBuckets.getCount(), splitTopicAndPartitionIndexLabel);
- }
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_overflow",
+ stats.compactionLatencyBuckets.getBuckets()[9], compactorMXBean, cluster, namespace, name,
+ splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_sum", stats.compactionLatencyBuckets.getSum(),
+ compactorMXBean, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_count", stats.compactionLatencyBuckets.getCount(),
+ compactorMXBean, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
}
- static void metricType(SimpleTextOutputStream stream, String name) {
-
- if (!metricWithTypeDefinition.containsKey(name)) {
- metricWithTypeDefinition.put(name, "gauge");
- stream.write("# TYPE ").write(name).write(" gauge\n");
- }
-
+ private static void writeMetric(Map<String, ByteBuf> allMetrics, String metricName, Number value, String cluster,
+ String namespace, String topic, boolean splitTopicAndPartitionIndexLabel) {
+ ByteBuf buffer = writeGaugeType(allMetrics, metricName);
+ writeTopicSample(buffer, metricName, value, cluster, namespace, topic, splitTopicAndPartitionIndexLabel);
}
- private static void metric(SimpleTextOutputStream stream, String cluster, String namespace, String topic,
- String name, double value, boolean splitTopicAndPartitionIndexLabel) {
- metricType(stream, name);
- appendRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel).write("\"} ");
- stream.write(value);
- appendEndings(stream);
+ private static void writeProducerStat(Map<String, ByteBuf> allMetrics, String metricName, TopicStats topicStats,
+ Function<AggregatedProducerStats, Number> valueFunction,
+ String cluster, String namespace, String topic,
+ boolean splitTopicAndPartitionIndexLabel) {
+ ByteBuf buffer = writeGaugeType(allMetrics, metricName);
+ topicStats.producerStats.forEach((p, producerStats) ->
+ writeTopicSample(buffer, metricName, valueFunction.apply(producerStats), cluster, namespace, topic,
+ splitTopicAndPartitionIndexLabel, "producer_name", p, "producer_id",
+ String.valueOf(producerStats.producerId)));
Review Comment:
@merlimat - how problematic is this? Should we do `producerStats.producerIdString` ?
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/TopicStats.java:
##########
@@ -102,380 +104,401 @@ public void reset() {
compactionLatencyBuckets.reset();
}
- static void resetTypes() {
- metricWithTypeDefinition.clear();
- }
-
- static void printTopicStats(SimpleTextOutputStream stream, String cluster, String namespace, String topic,
- TopicStats stats, Optional<CompactorMXBean> compactorMXBean,
- boolean splitTopicAndPartitionIndexLabel) {
- metric(stream, cluster, namespace, topic, "pulsar_subscriptions_count", stats.subscriptionsCount,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_producers_count", stats.producersCount,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_consumers_count", stats.consumersCount,
- splitTopicAndPartitionIndexLabel);
-
- metric(stream, cluster, namespace, topic, "pulsar_rate_in", stats.rateIn,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_rate_out", stats.rateOut,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_throughput_in", stats.throughputIn,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_throughput_out", stats.throughputOut,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_average_msg_size", stats.averageMsgSize,
- splitTopicAndPartitionIndexLabel);
-
- metric(stream, cluster, namespace, topic, "pulsar_storage_size", stats.managedLedgerStats.storageSize,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_logical_size",
- stats.managedLedgerStats.storageLogicalSize, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_msg_backlog", stats.msgBacklog,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_backlog_size",
- stats.managedLedgerStats.backlogSize, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_publish_rate_limit_times", stats.publishRateLimitedTimes,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_offloaded_size", stats.managedLedgerStats
- .offloadedStorageUsed, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_backlog_quota_limit", stats.backlogQuotaLimit,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_backlog_quota_limit_time",
- stats.backlogQuotaLimitTime, splitTopicAndPartitionIndexLabel);
-
- long[] latencyBuckets = stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets();
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_le_0_5", latencyBuckets[0],
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_le_1", latencyBuckets[1],
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_le_5", latencyBuckets[2],
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_le_10", latencyBuckets[3],
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_le_20", latencyBuckets[4],
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_le_50", latencyBuckets[5],
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_le_100", latencyBuckets[6],
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_le_200", latencyBuckets[7],
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_le_1000", latencyBuckets[8],
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_overflow", latencyBuckets[9],
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_count",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getCount(), splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_write_latency_sum",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getSum(), splitTopicAndPartitionIndexLabel);
-
- long[] ledgerWriteLatencyBuckets = stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets();
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_0_5",
- ledgerWriteLatencyBuckets[0], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_1",
- ledgerWriteLatencyBuckets[1], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_5",
- ledgerWriteLatencyBuckets[2], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_10",
- ledgerWriteLatencyBuckets[3], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_20",
- ledgerWriteLatencyBuckets[4], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_50",
- ledgerWriteLatencyBuckets[5], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_100",
- ledgerWriteLatencyBuckets[6], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_200",
- ledgerWriteLatencyBuckets[7], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_le_1000",
- ledgerWriteLatencyBuckets[8], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_overflow",
- ledgerWriteLatencyBuckets[9], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_count",
+ public static void printTopicStats(Map<String, ByteBuf> allMetrics, TopicStats stats,
+ Optional<CompactorMXBean> compactorMXBean, String cluster, String namespace,
+ String name, boolean splitTopicAndPartitionIndexLabel) {
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_subscriptions_count", stats.subscriptionsCount,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_producers_count", stats.producersCount,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_consumers_count", stats.consumersCount,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_rate_in", stats.rateIn,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_rate_out", stats.rateOut,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_throughput_in", stats.throughputIn,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_throughput_out", stats.throughputOut,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_average_msg_size", stats.averageMsgSize,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_storage_size", stats.managedLedgerStats.storageSize,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_storage_logical_size",
+ stats.managedLedgerStats.storageLogicalSize, cluster, namespace, name,
+ splitTopicAndPartitionIndexLabel);
+ writeMetricWithBrokerDefault(allMetrics, "pulsar_msg_backlog", stats.msgBacklog,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_backlog_size", stats.managedLedgerStats.backlogSize,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_publish_rate_limit_times", stats.publishRateLimitedTimes,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_offloaded_size", stats.managedLedgerStats
+ .offloadedStorageUsed, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_backlog_quota_limit", stats.backlogQuotaLimit,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_backlog_quota_limit_time", stats.backlogQuotaLimitTime,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_0_5",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[0],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_1",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[1],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_5",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[2],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_10",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[3],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_20",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[4],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_50",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[5],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_100",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[6],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_200",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[7],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_le_1000",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[8],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_overflow",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[9],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_count",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getCount(),
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_write_latency_sum",
+ stats.managedLedgerStats.storageWriteLatencyBuckets.getSum(), cluster, namespace, name,
+ splitTopicAndPartitionIndexLabel);
+
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_0_5",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[0],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_1",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[1],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_5",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[2],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_10",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[3],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_20",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[4],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_50",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[5],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_100",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[6],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_200",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[7],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_le_1000",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[8],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_overflow",
+ stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[9],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_count",
stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getCount(),
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_storage_ledger_write_latency_sum",
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_storage_ledger_write_latency_sum",
stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getSum(),
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+ writeMetric(allMetrics, "pulsar_entry_size_le_128",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[0],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_entry_size_le_512",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[1],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_entry_size_le_1_kb",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[2],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_entry_size_le_2_kb",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[3],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_entry_size_le_4_kb",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[4],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_entry_size_le_16_kb",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[5],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_entry_size_le_100_kb",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[6],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_entry_size_le_1_mb",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[7],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_entry_size_le_overflow",
+ stats.managedLedgerStats.entrySizeBuckets.getBuckets()[8],
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_entry_size_count", stats.managedLedgerStats.entrySizeBuckets.getCount(),
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_entry_size_sum", stats.managedLedgerStats.entrySizeBuckets.getSum(),
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+ writeProducerStat(allMetrics, "pulsar_producer_msg_rate_in", stats,
+ p -> p.msgRateIn, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeProducerStat(allMetrics, "pulsar_producer_msg_throughput_in", stats,
+ p -> p.msgThroughputIn, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeProducerStat(allMetrics, "pulsar_producer_msg_average_Size", stats,
+ p -> p.averageMsgSize, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_back_log", stats, s -> s.msgBacklog,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_back_log_no_delayed",
+ stats, s -> s.msgBacklogNoDelayed, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_delayed",
+ stats, s -> s.msgDelayed, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_msg_rate_redeliver",
+ stats, s -> s.msgRateRedeliver, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_unacked_messages",
+ stats, s -> s.unackedMessages, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_blocked_on_unacked_messages",
+ stats, s -> s.blockedSubscriptionOnUnackedMsgs ? 1 : 0, cluster, namespace, name,
+ splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_msg_rate_out",
+ stats, s -> s.msgRateOut, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_msg_ack_rate",
+ stats, s -> s.messageAckRate, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_msg_throughput_out",
+ stats, s -> s.msgThroughputOut, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_out_bytes_total",
+ stats, s -> s.bytesOutCounter, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_out_messages_total",
+ stats, s -> s.msgOutCounter, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_last_expire_timestamp",
+ stats, s -> s.lastExpireTimestamp, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_last_acked_timestamp",
+ stats, s -> s.lastAckedTimestamp, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_last_consumed_flow_timestamp",
+ stats, s -> s.lastConsumedFlowTimestamp, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_last_consumed_timestamp",
+ stats, s -> s.lastConsumedTimestamp, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_last_mark_delete_advanced_timestamp",
+ stats, s -> s.lastMarkDeleteAdvancedTimestamp, cluster, namespace, name,
+ splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_msg_rate_expired",
+ stats, s -> s.msgRateExpired, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_total_msg_expired",
+ stats, s -> s.totalMsgExpired, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_msg_drop_rate",
+ stats, s -> s.msgDropRate, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeSubscriptionStat(allMetrics, "pulsar_subscription_consumers_count",
+ stats, s -> s.consumersCount, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+ writeConsumerStat(allMetrics, "pulsar_consumer_msg_rate_redeliver", stats, c -> c.msgRateRedeliver,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeConsumerStat(allMetrics, "pulsar_consumer_unacked_messages", stats, c -> c.unackedMessages,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeConsumerStat(allMetrics, "pulsar_consumer_blocked_on_unacked_messages",
+ stats, c -> c.blockedSubscriptionOnUnackedMsgs ? 1 : 0,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeConsumerStat(allMetrics, "pulsar_consumer_msg_rate_out", stats, c -> c.msgRateOut,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+ writeConsumerStat(allMetrics, "pulsar_consumer_msg_ack_rate", stats, c -> c.msgAckRate,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+ writeConsumerStat(allMetrics, "pulsar_consumer_msg_throughput_out", stats, c -> c.msgThroughputOut,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeConsumerStat(allMetrics, "pulsar_consumer_available_permits", stats, c -> c.availablePermits,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeConsumerStat(allMetrics, "pulsar_out_bytes_total", stats, c -> c.bytesOutCounter,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeConsumerStat(allMetrics, "pulsar_out_messages_total", stats, c -> c.msgOutCounter,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+ writeReplicationStat(allMetrics, "pulsar_replication_rate_in", stats, r -> r.msgRateIn,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(allMetrics, "pulsar_replication_rate_out", stats, r -> r.msgRateOut,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(allMetrics, "pulsar_replication_throughput_in", stats, r -> r.msgThroughputIn,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(allMetrics, "pulsar_replication_throughput_out", stats, r -> r.msgThroughputOut,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(allMetrics, "pulsar_replication_backlog", stats, r -> r.replicationBacklog,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(allMetrics, "pulsar_replication_connected_count", stats, r -> r.connectedCount,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(allMetrics, "pulsar_replication_rate_expired", stats, r -> r.msgRateExpired,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeReplicationStat(allMetrics, "pulsar_replication_delay_in_seconds", stats,
+ r -> r.replicationDelayInSeconds, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+
+ writeMetric(allMetrics, "pulsar_in_bytes_total", stats.bytesInCounter, cluster, namespace, name,
+ splitTopicAndPartitionIndexLabel);
+ writeMetric(allMetrics, "pulsar_in_messages_total", stats.msgInCounter, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- long[] entrySizeBuckets = stats.managedLedgerStats.entrySizeBuckets.getBuckets();
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_le_128", entrySizeBuckets[0],
+ // Compaction
+
+ writeCompactionStat(allMetrics, "pulsar_compaction_removed_event_count", stats.compactionRemovedEventCount,
+ compactorMXBean, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(allMetrics, "pulsar_compaction_succeed_count", stats.compactionSucceedCount,
+ compactorMXBean, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(allMetrics, "pulsar_compaction_failed_count", stats.compactionFailedCount, compactorMXBean,
+ cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(allMetrics, "pulsar_compaction_duration_time_in_mills", stats.compactionDurationTimeInMills,
+ compactorMXBean, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(allMetrics, "pulsar_compaction_read_throughput", stats.compactionReadThroughput,
+ compactorMXBean, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(allMetrics, "pulsar_compaction_write_throughput", stats.compactionWriteThroughput,
+ compactorMXBean, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(allMetrics, "pulsar_compaction_compacted_entries_count",
+ stats.compactionCompactedEntriesCount, compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_le_512", entrySizeBuckets[1],
+ writeCompactionStat(allMetrics, "pulsar_compaction_compacted_entries_size",
+ stats.compactionCompactedEntriesSize, compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_le_1_kb", entrySizeBuckets[2],
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_le_0_5",
+ stats.compactionLatencyBuckets.getBuckets()[0], compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_le_2_kb", entrySizeBuckets[3],
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_le_1",
+ stats.compactionLatencyBuckets.getBuckets()[1], compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_le_4_kb", entrySizeBuckets[4],
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_le_5",
+ stats.compactionLatencyBuckets.getBuckets()[2], compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_le_16_kb", entrySizeBuckets[5],
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_le_10",
+ stats.compactionLatencyBuckets.getBuckets()[3], compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_le_100_kb", entrySizeBuckets[6],
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_le_20",
+ stats.compactionLatencyBuckets.getBuckets()[4], compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_le_1_mb", entrySizeBuckets[7],
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_le_50",
+ stats.compactionLatencyBuckets.getBuckets()[5], compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_le_overflow", entrySizeBuckets[8],
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_le_100",
+ stats.compactionLatencyBuckets.getBuckets()[6], compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_count",
- stats.managedLedgerStats.entrySizeBuckets.getCount(), splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_entry_size_sum",
- stats.managedLedgerStats.entrySizeBuckets.getSum(), splitTopicAndPartitionIndexLabel);
-
- stats.producerStats.forEach((p, producerStats) -> {
- metric(stream, cluster, namespace, topic, p, producerStats.producerId, "pulsar_producer_msg_rate_in",
- producerStats.msgRateIn, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, p, producerStats.producerId, "pulsar_producer_msg_throughput_in",
- producerStats.msgThroughputIn, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, p, producerStats.producerId, "pulsar_producer_msg_average_Size",
- producerStats.averageMsgSize, splitTopicAndPartitionIndexLabel);
- });
-
- stats.subscriptionStats.forEach((n, subsStats) -> {
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_back_log",
- subsStats.msgBacklog, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_back_log_no_delayed",
- subsStats.msgBacklogNoDelayed, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_delayed",
- subsStats.msgDelayed, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_msg_rate_redeliver",
- subsStats.msgRateRedeliver, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_unacked_messages",
- subsStats.unackedMessages, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_blocked_on_unacked_messages",
- subsStats.blockedSubscriptionOnUnackedMsgs ? 1 : 0, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_msg_rate_out",
- subsStats.msgRateOut, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_msg_ack_rate",
- subsStats.messageAckRate, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_msg_throughput_out",
- subsStats.msgThroughputOut, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_out_bytes_total",
- subsStats.bytesOutCounter, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_out_messages_total",
- subsStats.msgOutCounter, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_last_expire_timestamp",
- subsStats.lastExpireTimestamp, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_last_acked_timestamp",
- subsStats.lastAckedTimestamp, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_last_consumed_flow_timestamp",
- subsStats.lastConsumedFlowTimestamp, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_last_consumed_timestamp",
- subsStats.lastConsumedTimestamp, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_last_mark_delete_advanced_timestamp",
- subsStats.lastMarkDeleteAdvancedTimestamp, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_msg_rate_expired",
- subsStats.msgRateExpired, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_total_msg_expired",
- subsStats.totalMsgExpired, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_msg_drop_rate",
- subsStats.msgDropRate, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, "pulsar_subscription_consumers_count",
- subsStats.consumersCount, splitTopicAndPartitionIndexLabel);
- subsStats.consumerStat.forEach((c, consumerStats) -> {
- metric(stream, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_msg_rate_redeliver", consumerStats.msgRateRedeliver,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_unacked_messages", consumerStats.unackedMessages,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_blocked_on_unacked_messages",
- consumerStats.blockedSubscriptionOnUnackedMsgs ? 1 : 0,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_msg_rate_out", consumerStats.msgRateOut,
- splitTopicAndPartitionIndexLabel);
-
- metric(stream, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_msg_ack_rate", consumerStats.msgAckRate,
- splitTopicAndPartitionIndexLabel);
-
- metric(stream, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_msg_throughput_out", consumerStats.msgThroughputOut,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_consumer_available_permits", consumerStats.availablePermits,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_out_bytes_total", consumerStats.bytesOutCounter,
- splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, n, c.consumerName(), c.consumerId(),
- "pulsar_out_messages_total", consumerStats.msgOutCounter,
- splitTopicAndPartitionIndexLabel);
- });
- });
-
- if (!stats.replicationStats.isEmpty()) {
- stats.replicationStats.forEach((remoteCluster, replStats) -> {
- metricWithRemoteCluster(stream, cluster, namespace, topic, "pulsar_replication_rate_in", remoteCluster,
- replStats.msgRateIn, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(stream, cluster, namespace, topic, "pulsar_replication_rate_out", remoteCluster,
- replStats.msgRateOut, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(stream, cluster, namespace, topic, "pulsar_replication_throughput_in",
- remoteCluster, replStats.msgThroughputIn, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(stream, cluster, namespace, topic, "pulsar_replication_throughput_out",
- remoteCluster, replStats.msgThroughputOut, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(stream, cluster, namespace, topic, "pulsar_replication_backlog", remoteCluster,
- replStats.replicationBacklog, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(stream, cluster, namespace, topic, "pulsar_replication_connected_count",
- remoteCluster, replStats.connectedCount, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(stream, cluster, namespace, topic, "pulsar_replication_rate_expired",
- remoteCluster, replStats.msgRateExpired, splitTopicAndPartitionIndexLabel);
- metricWithRemoteCluster(stream, cluster, namespace, topic, "pulsar_replication_delay_in_seconds",
- remoteCluster, replStats.replicationDelayInSeconds, splitTopicAndPartitionIndexLabel);
- });
- }
-
- metric(stream, cluster, namespace, topic, "pulsar_in_bytes_total", stats.bytesInCounter,
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_le_200",
+ stats.compactionLatencyBuckets.getBuckets()[7], compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_in_messages_total", stats.msgInCounter,
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_le_1000",
+ stats.compactionLatencyBuckets.getBuckets()[8], compactorMXBean, cluster, namespace, name,
splitTopicAndPartitionIndexLabel);
-
- // Compaction
- boolean hasCompaction = compactorMXBean.flatMap(mxBean -> mxBean.getCompactionRecordForTopic(topic))
- .map(__ -> true).orElse(false);
- if (hasCompaction) {
- metric(stream, cluster, namespace, topic, "pulsar_compaction_removed_event_count",
- stats.compactionRemovedEventCount, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_succeed_count",
- stats.compactionSucceedCount, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_failed_count",
- stats.compactionFailedCount, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_duration_time_in_mills",
- stats.compactionDurationTimeInMills, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_read_throughput",
- stats.compactionReadThroughput, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_write_throughput",
- stats.compactionWriteThroughput, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_compacted_entries_count",
- stats.compactionCompactedEntriesCount, splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_compacted_entries_size",
- stats.compactionCompactedEntriesSize, splitTopicAndPartitionIndexLabel);
- long[] compactionLatencyBuckets = stats.compactionLatencyBuckets.getBuckets();
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_le_0_5",
- compactionLatencyBuckets[0], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_le_1",
- compactionLatencyBuckets[1], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_le_5",
- compactionLatencyBuckets[2], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_le_10",
- compactionLatencyBuckets[3], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_le_20",
- compactionLatencyBuckets[4], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_le_50",
- compactionLatencyBuckets[5], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_le_100",
- compactionLatencyBuckets[6], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_le_200",
- compactionLatencyBuckets[7], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_le_1000",
- compactionLatencyBuckets[8], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_overflow",
- compactionLatencyBuckets[9], splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_sum",
- stats.compactionLatencyBuckets.getSum(), splitTopicAndPartitionIndexLabel);
- metric(stream, cluster, namespace, topic, "pulsar_compaction_latency_count",
- stats.compactionLatencyBuckets.getCount(), splitTopicAndPartitionIndexLabel);
- }
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_overflow",
+ stats.compactionLatencyBuckets.getBuckets()[9], compactorMXBean, cluster, namespace, name,
+ splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_sum", stats.compactionLatencyBuckets.getSum(),
+ compactorMXBean, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
+ writeCompactionStat(allMetrics, "pulsar_compaction_latency_count", stats.compactionLatencyBuckets.getCount(),
+ compactorMXBean, cluster, namespace, name, splitTopicAndPartitionIndexLabel);
}
- static void metricType(SimpleTextOutputStream stream, String name) {
-
- if (!metricWithTypeDefinition.containsKey(name)) {
- metricWithTypeDefinition.put(name, "gauge");
- stream.write("# TYPE ").write(name).write(" gauge\n");
- }
-
+ private static void writeMetric(Map<String, ByteBuf> allMetrics, String metricName, Number value, String cluster,
+ String namespace, String topic, boolean splitTopicAndPartitionIndexLabel) {
+ ByteBuf buffer = writeGaugeType(allMetrics, metricName);
+ writeTopicSample(buffer, metricName, value, cluster, namespace, topic, splitTopicAndPartitionIndexLabel);
}
- private static void metric(SimpleTextOutputStream stream, String cluster, String namespace, String topic,
- String name, double value, boolean splitTopicAndPartitionIndexLabel) {
- metricType(stream, name);
- appendRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel).write("\"} ");
- stream.write(value);
- appendEndings(stream);
+ private static void writeProducerStat(Map<String, ByteBuf> allMetrics, String metricName, TopicStats topicStats,
+ Function<AggregatedProducerStats, Number> valueFunction,
+ String cluster, String namespace, String topic,
+ boolean splitTopicAndPartitionIndexLabel) {
+ ByteBuf buffer = writeGaugeType(allMetrics, metricName);
+ topicStats.producerStats.forEach((p, producerStats) ->
+ writeTopicSample(buffer, metricName, valueFunction.apply(producerStats), cluster, namespace, topic,
+ splitTopicAndPartitionIndexLabel, "producer_name", p, "producer_id",
+ String.valueOf(producerStats.producerId)));
}
- private static void metric(SimpleTextOutputStream stream, String cluster, String namespace, String topic,
- String subscription, String name, long value, boolean splitTopicAndPartitionIndexLabel) {
- metricType(stream, name);
- appendRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel)
- .write("\",subscription=\"").write(subscription).write("\"} ");
- stream.write(value);
- appendEndings(stream);
+ private static void writeSubscriptionStat(Map<String, ByteBuf> allMetrics, String metricName, TopicStats topicStats,
+ Function<AggregatedSubscriptionStats, Number> valueFunction,
+ String cluster, String namespace, String topic,
+ boolean splitTopicAndPartitionIndexLabel) {
+ ByteBuf buffer = writeGaugeType(allMetrics, metricName);
+ topicStats.subscriptionStats.forEach((s, subStats) ->
+ writeTopicSample(buffer, metricName, valueFunction.apply(subStats), cluster, namespace, topic,
+ splitTopicAndPartitionIndexLabel, "subscription", s));
}
- private static void metric(SimpleTextOutputStream stream, String cluster, String namespace, String topic,
- String producerName, long produceId, String name, double value, boolean splitTopicAndPartitionIndexLabel) {
- metricType(stream, name);
- appendRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel)
- .write("\",producer_name=\"").write(producerName)
- .write("\",producer_id=\"").write(produceId).write("\"} ");
- stream.write(value);
- appendEndings(stream);
+ private static void writeConsumerStat(Map<String, ByteBuf> allMetrics, String metricName, TopicStats topicStats,
+ Function<AggregatedConsumerStats, Number> valueFunction,
+ String cluster, String namespace, String topic,
+ boolean splitTopicAndPartitionIndexLabel) {
+ ByteBuf buffer = writeGaugeType(allMetrics, metricName);
+ topicStats.subscriptionStats.forEach((s, subStats) ->
+ subStats.consumerStat.forEach((c, conStats) ->
+ writeTopicSample(buffer, metricName, valueFunction.apply(conStats), cluster, namespace, topic,
+ splitTopicAndPartitionIndexLabel, "subscription", s, "consumer_name", c.consumerName(),
+ "consumer_id", String.valueOf(c.consumerId()))
+ ));
}
- private static void metric(SimpleTextOutputStream stream, String cluster, String namespace, String topic,
- String subscription, String name, double value, boolean splitTopicAndPartitionIndexLabel) {
- metricType(stream, name);
- appendRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel)
- .write("\",subscription=\"").write(subscription).write("\"} ");
- stream.write(value);
- appendEndings(stream);
- }
- private static void metric(SimpleTextOutputStream stream, String cluster, String namespace, String topic,
- String subscription, String consumerName, long consumerId, String name, long value,
- boolean splitTopicAndPartitionIndexLabel) {
- metricType(stream, name);
- appendRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel)
- .write("\",subscription=\"").write(subscription)
- .write("\",consumer_name=\"").write(consumerName).write("\",consumer_id=\"").write(consumerId)
- .write("\"} ");
- stream.write(value);
- appendEndings(stream);
+ private static void writeReplicationStat(Map<String, ByteBuf> allMetrics, String metricName, TopicStats topicStats,
+ Function<AggregatedReplicationStats, Number> valueFunction,
+ String cluster, String namespace, String topic,
+ boolean splitTopicAndPartitionIndexLabel) {
+ if (!topicStats.replicationStats.isEmpty()) {
+ ByteBuf buffer = writeGaugeType(allMetrics, metricName);
+ topicStats.replicationStats.forEach((remoteCluster, replStats) ->
+ writeTopicSample(buffer, metricName, valueFunction.apply(replStats), cluster, namespace, topic,
+ splitTopicAndPartitionIndexLabel, "remote_cluster", remoteCluster)
+ );
+ }
}
- private static void metric(SimpleTextOutputStream stream, String cluster, String namespace, String topic,
- String subscription, String consumerName, long consumerId, String name, double value,
- boolean splitTopicAndPartitionIndexLabel) {
- metricType(stream, name);
- appendRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel)
- .write("\",subscription=\"").write(subscription)
- .write("\",consumer_name=\"").write(consumerName).write("\",consumer_id=\"")
- .write(consumerId).write("\"} ");
- stream.write(value);
- appendEndings(stream);
+ private static void writeCompactionStat(Map<String, ByteBuf> allMetrics, String metricName,
+ Number value, Optional<CompactorMXBean> compactorMXBean,
+ String cluster, String namespace, String topic,
+ boolean splitTopicAndPartitionIndexLabel) {
+ boolean hasCompaction = compactorMXBean.flatMap(mxBean -> mxBean.getCompactionRecordForTopic(topic))
+ .isPresent();
+ if (hasCompaction) {
+ ByteBuf buffer = writeGaugeType(allMetrics, metricName);
+ writeTopicSample(buffer, metricName, value, cluster, namespace, topic, splitTopicAndPartitionIndexLabel);
+ }
}
- private static void metricWithRemoteCluster(SimpleTextOutputStream stream, String cluster, String namespace,
- String topic, String name, String remoteCluster, double value, boolean splitTopicAndPartitionIndexLabel) {
- metricType(stream, name);
- appendRequiredLabels(stream, cluster, namespace, topic, name, splitTopicAndPartitionIndexLabel)
- .write("\",remote_cluster=\"").write(remoteCluster).write("\"} ");
- stream.write(value);
- appendEndings(stream);
+ static void writeMetricWithBrokerDefault(Map<String, ByteBuf> allMetrics, String metricName, Number value,
+ String cluster, String namespace, String topic,
+ boolean splitTopicAndPartitionIndexLabel) {
+ ByteBuf buffer = writeGaugeTypeWithBrokerDefault(allMetrics, metricName, cluster);
+ writeTopicSample(buffer, metricName, value, cluster, namespace, topic, splitTopicAndPartitionIndexLabel);
}
- private static SimpleTextOutputStream appendRequiredLabels(SimpleTextOutputStream stream, String cluster,
- String namespace, String topic, String name, boolean splitTopicAndPartitionIndexLabel) {
- stream.write(name).write("{cluster=\"").write(cluster).write("\",namespace=\"").write(namespace);
+ static void writeTopicSample(ByteBuf buffer, String metricName, Number value, String cluster,
+ String namespace, String topic, boolean splitTopicAndPartitionIndexLabel,
+ String... extraLabelsAndValues) {
Review Comment:
@merlimat - is the suggestion I gave of using String[] and the usage of ArrayList ok? The granularity is per sample. I don't remember we have a load test, so we can know we if ruin performance or GC is ok with those short lived objects?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1216874652
@asafm Not sure I will have the bandwidth to look at `FunctionsStatsGenerator` anytime soon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
eolivelli commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1236950003
I don't think we should hurry and port too many "improvements" like this to branch-2.10
branch-2.10 must be kept stable and we should cherry-pick only stuff that we are sure it is not going to break stability or compatibility.
I don't think we need this patch on 2.10 or other older branches
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1129717933
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jason918 commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
Jason918 commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1229946170
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1235177811
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] asafm commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
asafm commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1216433375
@marksilcox I'm compiling a document detailing exactly how the metric system works today in Pulsar, and while doing so I've come to understand how Pulsar Function metrics work. The bad news: They also violate the grouping of the metric name. You will only experience it when you scrape Pulsar Function Worker process metrics or if you run it embedded within a Pulsar Broker.
The code is in `FunctionsStatsGenerator`.
So, the function there iterates over the different function runtimes. For example, if this worker runs function A with a replication factor of 2, and mode Process, it will spin up two operating system processes running `JavaInstanceStarter` which executes the function. Inside it runs a Prometheus HTTPServer which exposes its Prometheus metrics on a defined metrics port. Each process exposes the same metric name, so when FunctionsStatsGenerator aggregates them, it simply dumps one output into the same stream. This will probably require a separate PR with a different fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on a diff in pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on code in PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#discussion_r911809565
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/NamespaceStatsAggregator.java:
##########
@@ -305,155 +332,224 @@ private static void getTopicStats(Topic topic, TopicStats stats, boolean include
});
}
- private static void buildDefaultBrokerStats(Map<String, Collector.MetricFamilySamples> metrics, String cluster) {
- // Print metrics with 0 values. This is necessary to have the available brokers being
- // reported in the brokers dashboard even if they don't have any topic or traffic
- metric(metrics, cluster, "pulsar_topics_count", 0);
- metric(metrics, cluster, "pulsar_subscriptions_count", 0);
- metric(metrics, cluster, "pulsar_producers_count", 0);
- metric(metrics, cluster, "pulsar_consumers_count", 0);
- metric(metrics, cluster, "pulsar_rate_in", 0);
- metric(metrics, cluster, "pulsar_rate_out", 0);
- metric(metrics, cluster, "pulsar_throughput_in", 0);
- metric(metrics, cluster, "pulsar_throughput_out", 0);
- metric(metrics, cluster, "pulsar_storage_size", 0);
- metric(metrics, cluster, "pulsar_storage_logical_size", 0);
- metric(metrics, cluster, "pulsar_storage_write_rate", 0);
- metric(metrics, cluster, "pulsar_storage_read_rate", 0);
- metric(metrics, cluster, "pulsar_msg_backlog", 0);
+ private static void printTopicsCountStats(SimpleTextOutputStream stream, String cluster,
+ Map<String, Long> namespaceTopicsCount) {
+ stream.write("# TYPE ").write("pulsar_topics_count").write(" gauge\n");
+ stream.write("pulsar_topics_count")
+ .write("{cluster=\"").write(cluster).write("\"} ")
+ .write(0).write(' ').write(System.currentTimeMillis())
+ .write('\n');
+ namespaceTopicsCount.forEach((ns, topicCount) -> stream.write("pulsar_topics_count")
+ .write("{cluster=\"").write(cluster)
+ .write("\",namespace=\"").write(ns)
+ .write("\"} ")
+ .write(topicCount).write(' ').write(System.currentTimeMillis())
+ .write('\n')
+ );
}
- private static void printTopicsCountStats(Map<String, Collector.MetricFamilySamples> metrics, String cluster,
- String namespace,
- LongAdder topicsCount) {
- metric(metrics, cluster, namespace, "pulsar_topics_count", topicsCount.sum());
+ private static void printNamespaceStats(SimpleTextOutputStream stream, String cluster,
+ List<AggregatedNamespaceStats> stats) {
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_topics_count", stats, s -> s.topicsCount);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_subscriptions_count", stats, s -> s.subscriptionsCount);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_producers_count", stats, s -> s.producersCount);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_consumers_count", stats, s -> s.consumersCount);
+
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_rate_in", stats, s -> s.rateIn);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_rate_out", stats, s -> s.rateOut);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_throughput_in", stats, s -> s.throughputIn);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_throughput_out", stats, s -> s.throughputOut);
+ writeMetric(stream, cluster, "pulsar_consumer_msg_ack_rate", stats, s -> s.messageAckRate);
+
+ writeMetric(stream, cluster, "pulsar_in_bytes_total", stats, s -> s.bytesInCounter);
+ writeMetric(stream, cluster, "pulsar_in_messages_total", stats, s -> s.msgInCounter);
+ writeMetric(stream, cluster, "pulsar_out_bytes_total", stats, s -> s.bytesOutCounter);
+ writeMetric(stream, cluster, "pulsar_out_messages_total", stats, s -> s.msgOutCounter);
+
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_size", stats,
+ s -> s.managedLedgerStats.storageSize);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_logical_size", stats,
+ s -> s.managedLedgerStats.storageLogicalSize);
+ writeMetric(stream, cluster, "pulsar_storage_backlog_size", stats, s -> s.managedLedgerStats.backlogSize);
+ writeMetric(stream, cluster, "pulsar_storage_offloaded_size",
+ stats, s -> s.managedLedgerStats.offloadedStorageUsed);
+
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_write_rate", stats,
+ s -> s.managedLedgerStats.storageWriteRate);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_read_rate", stats,
+ s -> s.managedLedgerStats.storageReadRate);
+
+ writeMetric(stream, cluster, "pulsar_subscription_delayed", stats, s -> s.msgDelayed);
+
+ writeMsgBacklog(stream, cluster, stats, s -> s.msgBacklog);
+
+ stats.forEach(s -> s.managedLedgerStats.storageWriteLatencyBuckets.refresh());
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_0_5", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[0]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_1", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[1]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_5", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[2]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_10", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[3]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_20", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[4]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_50", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[5]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_100", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[6]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_200", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[7]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_1000", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[8]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_overflow", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[9]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_count",
+ stats, s -> s.managedLedgerStats.storageWriteLatencyBuckets.getCount());
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_sum",
+ stats, s -> s.managedLedgerStats.storageWriteLatencyBuckets.getSum());
+
+ stats.forEach(s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.refresh());
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_0_5", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[0]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_1", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[1]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_5", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[2]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_10", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[3]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_20", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[4]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_50", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[5]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_100", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[6]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_200", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[7]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_1000",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[8]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_overflow",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[9]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_count",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getCount());
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_sum",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getSum());
+
+ stats.forEach(s -> s.managedLedgerStats.entrySizeBuckets.refresh());
+ writeMetric(stream, cluster, "pulsar_entry_size_le_128", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[0]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_512", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[1]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_1_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[2]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_2_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[3]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_4_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[4]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_16_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[5]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_100_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[6]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_1_mb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[7]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_overflow", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[8]);
+ writeMetric(stream, cluster, "pulsar_entry_size_count",
+ stats, s -> s.managedLedgerStats.entrySizeBuckets.getCount());
+ writeMetric(stream, cluster, "pulsar_entry_size_sum",
+ stats, s -> s.managedLedgerStats.entrySizeBuckets.getSum());
+
+ writeReplicationStat(stream, cluster, "pulsar_replication_rate_in", stats,
+ replStats -> replStats.msgRateIn);
+ writeReplicationStat(stream, cluster, "pulsar_replication_rate_out", stats,
+ replStats -> replStats.msgRateOut);
+ writeReplicationStat(stream, cluster, "pulsar_replication_throughput_in", stats,
+ replStats -> replStats.msgThroughputIn);
+ writeReplicationStat(stream, cluster, "pulsar_replication_throughput_out", stats,
+ replStats -> replStats.msgThroughputOut);
+ writeReplicationStat(stream, cluster, "pulsar_replication_backlog", stats,
+ replStats -> replStats.replicationBacklog);
+ writeReplicationStat(stream, cluster, "pulsar_replication_connected_count", stats,
+ replStats -> replStats.connectedCount);
+ writeReplicationStat(stream, cluster, "pulsar_replication_rate_expired", stats,
+ replStats -> replStats.msgRateExpired);
+ writeReplicationStat(stream, cluster, "pulsar_replication_delay_in_seconds", stats,
+ replStats -> replStats.replicationDelayInSeconds);
}
- private static void printNamespaceStats(Map<String, Collector.MetricFamilySamples> metrics, String cluster,
- String namespace,
- AggregatedNamespaceStats stats) {
- metric(metrics, cluster, namespace, "pulsar_topics_count", stats.topicsCount);
- metric(metrics, cluster, namespace, "pulsar_subscriptions_count", stats.subscriptionsCount);
- metric(metrics, cluster, namespace, "pulsar_producers_count", stats.producersCount);
- metric(metrics, cluster, namespace, "pulsar_consumers_count", stats.consumersCount);
-
- metric(metrics, cluster, namespace, "pulsar_rate_in", stats.rateIn);
- metric(metrics, cluster, namespace, "pulsar_rate_out", stats.rateOut);
- metric(metrics, cluster, namespace, "pulsar_throughput_in", stats.throughputIn);
- metric(metrics, cluster, namespace, "pulsar_throughput_out", stats.throughputOut);
- metric(metrics, cluster, namespace, "pulsar_consumer_msg_ack_rate", stats.messageAckRate);
-
- metric(metrics, cluster, namespace, "pulsar_in_bytes_total", stats.bytesInCounter);
- metric(metrics, cluster, namespace, "pulsar_in_messages_total", stats.msgInCounter);
- metric(metrics, cluster, namespace, "pulsar_out_bytes_total", stats.bytesOutCounter);
- metric(metrics, cluster, namespace, "pulsar_out_messages_total", stats.msgOutCounter);
-
- metric(metrics, cluster, namespace, "pulsar_storage_size", stats.managedLedgerStats.storageSize);
- metric(metrics, cluster, namespace, "pulsar_storage_logical_size", stats.managedLedgerStats.storageLogicalSize);
- metric(metrics, cluster, namespace, "pulsar_storage_backlog_size", stats.managedLedgerStats.backlogSize);
- metric(metrics, cluster, namespace, "pulsar_storage_offloaded_size",
- stats.managedLedgerStats.offloadedStorageUsed);
-
- metric(metrics, cluster, namespace, "pulsar_storage_write_rate", stats.managedLedgerStats.storageWriteRate);
- metric(metrics, cluster, namespace, "pulsar_storage_read_rate", stats.managedLedgerStats.storageReadRate);
-
- metric(metrics, cluster, namespace, "pulsar_subscription_delayed", stats.msgDelayed);
-
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_msg_backlog", "local", stats.msgBacklog);
-
- stats.managedLedgerStats.storageWriteLatencyBuckets.refresh();
- long[] latencyBuckets = stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets();
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_0_5", latencyBuckets[0]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_1", latencyBuckets[1]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_5", latencyBuckets[2]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_10", latencyBuckets[3]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_20", latencyBuckets[4]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_50", latencyBuckets[5]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_100", latencyBuckets[6]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_200", latencyBuckets[7]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_1000", latencyBuckets[8]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_overflow", latencyBuckets[9]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_count",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getCount());
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_sum",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getSum());
-
- stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.refresh();
- long[] ledgerWritelatencyBuckets = stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets();
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_0_5", ledgerWritelatencyBuckets[0]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_1", ledgerWritelatencyBuckets[1]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_5", ledgerWritelatencyBuckets[2]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_10", ledgerWritelatencyBuckets[3]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_20", ledgerWritelatencyBuckets[4]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_50", ledgerWritelatencyBuckets[5]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_100", ledgerWritelatencyBuckets[6]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_200", ledgerWritelatencyBuckets[7]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_1000",
- ledgerWritelatencyBuckets[8]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_overflow",
- ledgerWritelatencyBuckets[9]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_count",
- stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getCount());
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_sum",
- stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getSum());
-
- stats.managedLedgerStats.entrySizeBuckets.refresh();
- long[] entrySizeBuckets = stats.managedLedgerStats.entrySizeBuckets.getBuckets();
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_128", entrySizeBuckets[0]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_512", entrySizeBuckets[1]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_1_kb", entrySizeBuckets[2]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_2_kb", entrySizeBuckets[3]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_4_kb", entrySizeBuckets[4]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_16_kb", entrySizeBuckets[5]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_100_kb", entrySizeBuckets[6]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_1_mb", entrySizeBuckets[7]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_overflow", entrySizeBuckets[8]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_count",
- stats.managedLedgerStats.entrySizeBuckets.getCount());
- metric(metrics, cluster, namespace, "pulsar_entry_size_sum",
- stats.managedLedgerStats.entrySizeBuckets.getSum());
-
- if (!stats.replicationStats.isEmpty()) {
- stats.replicationStats.forEach((remoteCluster, replStats) -> {
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_rate_in", remoteCluster,
- replStats.msgRateIn);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_rate_out", remoteCluster,
- replStats.msgRateOut);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_throughput_in", remoteCluster,
- replStats.msgThroughputIn);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_throughput_out", remoteCluster,
- replStats.msgThroughputOut);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_backlog", remoteCluster,
- replStats.replicationBacklog);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_connected_count",
- remoteCluster,
- replStats.connectedCount);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_rate_expired", remoteCluster,
- replStats.msgRateExpired);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_delay_in_seconds",
- remoteCluster, replStats.replicationDelayInSeconds);
- });
- }
+ private static void writeMetricWithBrokerDefault(SimpleTextOutputStream stream, String cluster, String name,
+ List<AggregatedNamespaceStats> allNamespaceStats,
+ Function<AggregatedNamespaceStats, Number> namespaceFunction) {
+ stream.write("# TYPE ").write(name).write(" gauge\n");
+ stream.write(name)
+ .write("{cluster=\"").write(cluster).write("\"} ")
+ .write(0).write(' ').write(System.currentTimeMillis())
+ .write('\n');
+ writeNamespaceStats(stream, cluster, name, allNamespaceStats, namespaceFunction);
}
- private static void metric(Map<String, Collector.MetricFamilySamples> metrics, String cluster, String name,
- long value) {
- addMetric(metrics, Map.of("cluster", cluster), name, value);
+ private static void writeMetric(SimpleTextOutputStream stream, String cluster, String name,
+ List<AggregatedNamespaceStats> allNamespaceStats,
+ Function<AggregatedNamespaceStats, Number> namespaceFunction) {
+ stream.write("# TYPE ").write(name).write(" gauge\n");
+ writeNamespaceStats(stream, cluster, name, allNamespaceStats, namespaceFunction);
}
- private static void metric(Map<String, Collector.MetricFamilySamples> metrics, String cluster, String namespace,
- String name, long value) {
- addMetric(metrics, Map.of("cluster", cluster, "namespace", namespace), name, value);
+ private static void writeNamespaceStats(SimpleTextOutputStream stream, String cluster, String name,
+ List<AggregatedNamespaceStats> allNamespaceStats,
+ Function<AggregatedNamespaceStats, Number> namespaceFunction) {
+ allNamespaceStats.forEach(n -> {
+ Number value = namespaceFunction.apply(n);
+ if (value != null) {
+ stream.write(name)
+ .write("{cluster=\"").write(cluster)
+ .write("\",namespace=\"").write(n.name)
+ .write("\"} ")
+ .write(value).write(' ').write(System.currentTimeMillis())
+ .write('\n');
+ }
+ });
}
- private static void metric(Map<String, Collector.MetricFamilySamples> metrics, String cluster, String namespace,
- String name, double value) {
- addMetric(metrics, Map.of("cluster", cluster, "namespace", namespace), name, value);
+ private static void writeMsgBacklog(SimpleTextOutputStream stream, String cluster,
+ List<AggregatedNamespaceStats> allNamespaceStats,
+ Function<AggregatedNamespaceStats, Number> namespaceFunction) {
+ stream.write("# TYPE ").write("pulsar_msg_backlog").write(" gauge\n");
+ stream.write("pulsar_msg_backlog")
Review Comment:
This was originally written in `printDefaultBrokerStats` which was always called.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1162000349
@asafm
I may be missing something, but to be able to print the metrics in the Prometheus format we would still need to collate all TopicStats and all AggregatedNamespaceStats as they share metric types (e.g. there is a `pulsar_rate_in` metric for both the namespace and all topics in that namespace). Then aggregate across all by metric name.
There are usages elsewhere in Pulsar of MetricFamilySamples, which is why I went down this route rather than creating a new model to hold these relationships.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] asafm commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
asafm commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1162166635
NamespaceStats and TopicStat do share metrics as they use the same metrics but aggregate to different level. One is raw, at topic level, and the other is simple aggregation to namespace level.
The includeTopics flag is actually an either: includeTopics actually means, print only topic level metrics, while false means print only namespace level metrics. So there's no collision then no?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] tjiuming commented on pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
tjiuming commented on PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#issuecomment-1161704633
@asafm I thought of another way of grouping these metrics, how about write metrics-data into a same ByteBuf which has the same metric name?
In the finally, merge these ByteBufs into `stream`, seems it won't takes too much costs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on a diff in pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on code in PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#discussion_r911822912
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/NamespaceStatsAggregator.java:
##########
@@ -305,155 +332,224 @@ private static void getTopicStats(Topic topic, TopicStats stats, boolean include
});
}
- private static void buildDefaultBrokerStats(Map<String, Collector.MetricFamilySamples> metrics, String cluster) {
- // Print metrics with 0 values. This is necessary to have the available brokers being
- // reported in the brokers dashboard even if they don't have any topic or traffic
- metric(metrics, cluster, "pulsar_topics_count", 0);
- metric(metrics, cluster, "pulsar_subscriptions_count", 0);
- metric(metrics, cluster, "pulsar_producers_count", 0);
- metric(metrics, cluster, "pulsar_consumers_count", 0);
- metric(metrics, cluster, "pulsar_rate_in", 0);
- metric(metrics, cluster, "pulsar_rate_out", 0);
- metric(metrics, cluster, "pulsar_throughput_in", 0);
- metric(metrics, cluster, "pulsar_throughput_out", 0);
- metric(metrics, cluster, "pulsar_storage_size", 0);
- metric(metrics, cluster, "pulsar_storage_logical_size", 0);
- metric(metrics, cluster, "pulsar_storage_write_rate", 0);
- metric(metrics, cluster, "pulsar_storage_read_rate", 0);
- metric(metrics, cluster, "pulsar_msg_backlog", 0);
+ private static void printTopicsCountStats(SimpleTextOutputStream stream, String cluster,
+ Map<String, Long> namespaceTopicsCount) {
+ stream.write("# TYPE ").write("pulsar_topics_count").write(" gauge\n");
+ stream.write("pulsar_topics_count")
+ .write("{cluster=\"").write(cluster).write("\"} ")
+ .write(0).write(' ').write(System.currentTimeMillis())
+ .write('\n');
+ namespaceTopicsCount.forEach((ns, topicCount) -> stream.write("pulsar_topics_count")
+ .write("{cluster=\"").write(cluster)
+ .write("\",namespace=\"").write(ns)
+ .write("\"} ")
+ .write(topicCount).write(' ').write(System.currentTimeMillis())
+ .write('\n')
+ );
}
- private static void printTopicsCountStats(Map<String, Collector.MetricFamilySamples> metrics, String cluster,
- String namespace,
- LongAdder topicsCount) {
- metric(metrics, cluster, namespace, "pulsar_topics_count", topicsCount.sum());
+ private static void printNamespaceStats(SimpleTextOutputStream stream, String cluster,
+ List<AggregatedNamespaceStats> stats) {
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_topics_count", stats, s -> s.topicsCount);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_subscriptions_count", stats, s -> s.subscriptionsCount);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_producers_count", stats, s -> s.producersCount);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_consumers_count", stats, s -> s.consumersCount);
+
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_rate_in", stats, s -> s.rateIn);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_rate_out", stats, s -> s.rateOut);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_throughput_in", stats, s -> s.throughputIn);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_throughput_out", stats, s -> s.throughputOut);
+ writeMetric(stream, cluster, "pulsar_consumer_msg_ack_rate", stats, s -> s.messageAckRate);
+
+ writeMetric(stream, cluster, "pulsar_in_bytes_total", stats, s -> s.bytesInCounter);
+ writeMetric(stream, cluster, "pulsar_in_messages_total", stats, s -> s.msgInCounter);
+ writeMetric(stream, cluster, "pulsar_out_bytes_total", stats, s -> s.bytesOutCounter);
+ writeMetric(stream, cluster, "pulsar_out_messages_total", stats, s -> s.msgOutCounter);
+
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_size", stats,
+ s -> s.managedLedgerStats.storageSize);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_logical_size", stats,
+ s -> s.managedLedgerStats.storageLogicalSize);
+ writeMetric(stream, cluster, "pulsar_storage_backlog_size", stats, s -> s.managedLedgerStats.backlogSize);
+ writeMetric(stream, cluster, "pulsar_storage_offloaded_size",
+ stats, s -> s.managedLedgerStats.offloadedStorageUsed);
+
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_write_rate", stats,
+ s -> s.managedLedgerStats.storageWriteRate);
+ writeMetricWithBrokerDefault(stream, cluster, "pulsar_storage_read_rate", stats,
+ s -> s.managedLedgerStats.storageReadRate);
+
+ writeMetric(stream, cluster, "pulsar_subscription_delayed", stats, s -> s.msgDelayed);
+
+ writeMsgBacklog(stream, cluster, stats, s -> s.msgBacklog);
+
+ stats.forEach(s -> s.managedLedgerStats.storageWriteLatencyBuckets.refresh());
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_0_5", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[0]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_1", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[1]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_5", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[2]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_10", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[3]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_20", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[4]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_50", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[5]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_100", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[6]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_200", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[7]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_le_1000", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[8]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_overflow", stats,
+ s -> s.managedLedgerStats.storageWriteLatencyBuckets.getBuckets()[9]);
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_count",
+ stats, s -> s.managedLedgerStats.storageWriteLatencyBuckets.getCount());
+ writeMetric(stream, cluster, "pulsar_storage_write_latency_sum",
+ stats, s -> s.managedLedgerStats.storageWriteLatencyBuckets.getSum());
+
+ stats.forEach(s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.refresh());
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_0_5", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[0]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_1", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[1]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_5", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[2]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_10", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[3]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_20", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[4]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_50", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[5]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_100", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[6]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_200", stats,
+ s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[7]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_le_1000",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[8]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_overflow",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets()[9]);
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_count",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getCount());
+ writeMetric(stream, cluster, "pulsar_storage_ledger_write_latency_sum",
+ stats, s -> s.managedLedgerStats.storageLedgerWriteLatencyBuckets.getSum());
+
+ stats.forEach(s -> s.managedLedgerStats.entrySizeBuckets.refresh());
+ writeMetric(stream, cluster, "pulsar_entry_size_le_128", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[0]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_512", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[1]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_1_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[2]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_2_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[3]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_4_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[4]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_16_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[5]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_100_kb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[6]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_1_mb", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[7]);
+ writeMetric(stream, cluster, "pulsar_entry_size_le_overflow", stats,
+ s -> s.managedLedgerStats.entrySizeBuckets.getBuckets()[8]);
+ writeMetric(stream, cluster, "pulsar_entry_size_count",
+ stats, s -> s.managedLedgerStats.entrySizeBuckets.getCount());
+ writeMetric(stream, cluster, "pulsar_entry_size_sum",
+ stats, s -> s.managedLedgerStats.entrySizeBuckets.getSum());
+
+ writeReplicationStat(stream, cluster, "pulsar_replication_rate_in", stats,
+ replStats -> replStats.msgRateIn);
+ writeReplicationStat(stream, cluster, "pulsar_replication_rate_out", stats,
+ replStats -> replStats.msgRateOut);
+ writeReplicationStat(stream, cluster, "pulsar_replication_throughput_in", stats,
+ replStats -> replStats.msgThroughputIn);
+ writeReplicationStat(stream, cluster, "pulsar_replication_throughput_out", stats,
+ replStats -> replStats.msgThroughputOut);
+ writeReplicationStat(stream, cluster, "pulsar_replication_backlog", stats,
+ replStats -> replStats.replicationBacklog);
+ writeReplicationStat(stream, cluster, "pulsar_replication_connected_count", stats,
+ replStats -> replStats.connectedCount);
+ writeReplicationStat(stream, cluster, "pulsar_replication_rate_expired", stats,
+ replStats -> replStats.msgRateExpired);
+ writeReplicationStat(stream, cluster, "pulsar_replication_delay_in_seconds", stats,
+ replStats -> replStats.replicationDelayInSeconds);
}
- private static void printNamespaceStats(Map<String, Collector.MetricFamilySamples> metrics, String cluster,
- String namespace,
- AggregatedNamespaceStats stats) {
- metric(metrics, cluster, namespace, "pulsar_topics_count", stats.topicsCount);
- metric(metrics, cluster, namespace, "pulsar_subscriptions_count", stats.subscriptionsCount);
- metric(metrics, cluster, namespace, "pulsar_producers_count", stats.producersCount);
- metric(metrics, cluster, namespace, "pulsar_consumers_count", stats.consumersCount);
-
- metric(metrics, cluster, namespace, "pulsar_rate_in", stats.rateIn);
- metric(metrics, cluster, namespace, "pulsar_rate_out", stats.rateOut);
- metric(metrics, cluster, namespace, "pulsar_throughput_in", stats.throughputIn);
- metric(metrics, cluster, namespace, "pulsar_throughput_out", stats.throughputOut);
- metric(metrics, cluster, namespace, "pulsar_consumer_msg_ack_rate", stats.messageAckRate);
-
- metric(metrics, cluster, namespace, "pulsar_in_bytes_total", stats.bytesInCounter);
- metric(metrics, cluster, namespace, "pulsar_in_messages_total", stats.msgInCounter);
- metric(metrics, cluster, namespace, "pulsar_out_bytes_total", stats.bytesOutCounter);
- metric(metrics, cluster, namespace, "pulsar_out_messages_total", stats.msgOutCounter);
-
- metric(metrics, cluster, namespace, "pulsar_storage_size", stats.managedLedgerStats.storageSize);
- metric(metrics, cluster, namespace, "pulsar_storage_logical_size", stats.managedLedgerStats.storageLogicalSize);
- metric(metrics, cluster, namespace, "pulsar_storage_backlog_size", stats.managedLedgerStats.backlogSize);
- metric(metrics, cluster, namespace, "pulsar_storage_offloaded_size",
- stats.managedLedgerStats.offloadedStorageUsed);
-
- metric(metrics, cluster, namespace, "pulsar_storage_write_rate", stats.managedLedgerStats.storageWriteRate);
- metric(metrics, cluster, namespace, "pulsar_storage_read_rate", stats.managedLedgerStats.storageReadRate);
-
- metric(metrics, cluster, namespace, "pulsar_subscription_delayed", stats.msgDelayed);
-
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_msg_backlog", "local", stats.msgBacklog);
-
- stats.managedLedgerStats.storageWriteLatencyBuckets.refresh();
- long[] latencyBuckets = stats.managedLedgerStats.storageWriteLatencyBuckets.getBuckets();
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_0_5", latencyBuckets[0]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_1", latencyBuckets[1]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_5", latencyBuckets[2]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_10", latencyBuckets[3]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_20", latencyBuckets[4]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_50", latencyBuckets[5]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_100", latencyBuckets[6]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_200", latencyBuckets[7]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_le_1000", latencyBuckets[8]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_overflow", latencyBuckets[9]);
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_count",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getCount());
- metric(metrics, cluster, namespace, "pulsar_storage_write_latency_sum",
- stats.managedLedgerStats.storageWriteLatencyBuckets.getSum());
-
- stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.refresh();
- long[] ledgerWritelatencyBuckets = stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getBuckets();
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_0_5", ledgerWritelatencyBuckets[0]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_1", ledgerWritelatencyBuckets[1]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_5", ledgerWritelatencyBuckets[2]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_10", ledgerWritelatencyBuckets[3]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_20", ledgerWritelatencyBuckets[4]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_50", ledgerWritelatencyBuckets[5]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_100", ledgerWritelatencyBuckets[6]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_200", ledgerWritelatencyBuckets[7]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_le_1000",
- ledgerWritelatencyBuckets[8]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_overflow",
- ledgerWritelatencyBuckets[9]);
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_count",
- stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getCount());
- metric(metrics, cluster, namespace, "pulsar_storage_ledger_write_latency_sum",
- stats.managedLedgerStats.storageLedgerWriteLatencyBuckets.getSum());
-
- stats.managedLedgerStats.entrySizeBuckets.refresh();
- long[] entrySizeBuckets = stats.managedLedgerStats.entrySizeBuckets.getBuckets();
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_128", entrySizeBuckets[0]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_512", entrySizeBuckets[1]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_1_kb", entrySizeBuckets[2]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_2_kb", entrySizeBuckets[3]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_4_kb", entrySizeBuckets[4]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_16_kb", entrySizeBuckets[5]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_100_kb", entrySizeBuckets[6]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_1_mb", entrySizeBuckets[7]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_le_overflow", entrySizeBuckets[8]);
- metric(metrics, cluster, namespace, "pulsar_entry_size_count",
- stats.managedLedgerStats.entrySizeBuckets.getCount());
- metric(metrics, cluster, namespace, "pulsar_entry_size_sum",
- stats.managedLedgerStats.entrySizeBuckets.getSum());
-
- if (!stats.replicationStats.isEmpty()) {
- stats.replicationStats.forEach((remoteCluster, replStats) -> {
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_rate_in", remoteCluster,
- replStats.msgRateIn);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_rate_out", remoteCluster,
- replStats.msgRateOut);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_throughput_in", remoteCluster,
- replStats.msgThroughputIn);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_throughput_out", remoteCluster,
- replStats.msgThroughputOut);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_backlog", remoteCluster,
- replStats.replicationBacklog);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_connected_count",
- remoteCluster,
- replStats.connectedCount);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_rate_expired", remoteCluster,
- replStats.msgRateExpired);
- metricWithRemoteCluster(metrics, cluster, namespace, "pulsar_replication_delay_in_seconds",
- remoteCluster, replStats.replicationDelayInSeconds);
- });
- }
+ private static void writeMetricWithBrokerDefault(SimpleTextOutputStream stream, String cluster, String name,
+ List<AggregatedNamespaceStats> allNamespaceStats,
+ Function<AggregatedNamespaceStats, Number> namespaceFunction) {
+ stream.write("# TYPE ").write(name).write(" gauge\n");
+ stream.write(name)
Review Comment:
Current behaviour is to always write
``` java
private static void printDefaultBrokerStats(SimpleTextOutputStream stream, String cluster) {
// Print metrics with 0 values. This is necessary to have the available brokers being
// reported in the brokers dashboard even if they don't have any topic or traffic
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] marksilcox commented on a diff in pull request #15558: [fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)
Posted by GitBox <gi...@apache.org>.
marksilcox commented on code in PR #15558:
URL: https://github.com/apache/pulsar/pull/15558#discussion_r914831642
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricStreams.java:
##########
@@ -0,0 +1,74 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pulsar.broker.stats.prometheus;
+
+import io.netty.buffer.ByteBufAllocator;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.pulsar.common.util.SimpleTextOutputStream;
+
+/**
+ * Helper class to ensure that metrics of the same name are grouped together under the same TYPE header when written.
+ */
+public class PrometheusMetricStreams {
+ private final Map<String, SimpleTextOutputStream> metricStreamMap = new HashMap<>();
+
+ /**
+ * Write the given metric and sample value to the stream. Will write #TYPE header if metric not seen before.
+ * @param metricName name of the metric.
+ * @param value value of the sample
+ * @param labelsAndValuesArray varargs of label and label value
+ */
+ void writeSample(String metricName, Number value, String... labelsAndValuesArray) {
+ SimpleTextOutputStream stream = initGaugeType(metricName);
+ stream.write(metricName).write('{');
+ for (int i = 0; i < labelsAndValuesArray.length; i += 2) {
+ stream.write(labelsAndValuesArray[i]).write("=\"").write(labelsAndValuesArray[i + 1]).write('\"');
+ if (i + 2 != labelsAndValuesArray.length) {
+ stream.write(',');
+ }
+ }
+ stream.write("\"} ").write(value).write(' ').write(System.currentTimeMillis()).write('\n');
Review Comment:
time was written in the original code - see `TopicStats#appendEndings` which is used in all topic and namespace writing, also all metric methods in `TransactionAggregator`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org