You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "dragosvictor (via GitHub)" <gi...@apache.org> on 2024/06/07 23:40:02 UTC

[PR] [feat][doc] PIP-264: Add OpenTelemetry producer metrics reference [pulsar-site]

dragosvictor opened a new pull request, #911:
URL: https://github.com/apache/pulsar-site/pull/911

   ### ✅ Contribution Checklist
   
   - [x] I read the [contribution guide](https://pulsar.apache.org/contribute/document-contribution/)
   - [x] I updated the [versioned docs](https://pulsar.apache.org/contribute/document-contribution/#update-versioned-docs)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [feat][doc] PIP-264: Add OpenTelemetry producer metrics reference [pulsar-site]

Posted by "heesung-sn (via GitHub)" <gi...@apache.org>.
heesung-sn merged PR #911:
URL: https://github.com/apache/pulsar-site/pull/911


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [feat][doc] PIP-264: Add OpenTelemetry producer metrics reference [pulsar-site]

Posted by "dragosvictor (via GitHub)" <gi...@apache.org>.
dragosvictor commented on code in PR #911:
URL: https://github.com/apache/pulsar-site/pull/911#discussion_r1636947032


##########
docs/reference-metrics-opentelemetry.md:
##########
@@ -402,3 +402,62 @@ The number of pending topic load operations in the broker. When it reaches thres
 The maximum number of pending topic load operations in the broker. Equal to "maxConcurrentTopicLoadRequest" defined in broker.conf.
 * Type: UpDownCounter
 * Unit: `{operation}`
+
+### Producer metrics
+
+#### pulsar.broker.producer.message.incoming.count
+The total number of messages received from this producer.
+* Type: Counter
+* Unit: `{message}`
+* Attributes:

Review Comment:
   Yes, cluster information is automatically included in all the broker metrics, as part of the OpenTelemetry resource attribute set. There are many resource attributes attached to the metrics emitted by the broker, so I avoided repeating them everywhere. You can see the ones Pulsar itself is adding here: https://github.com/apache/pulsar/blob/c724f02e6ab4f342e805b21cc99c394b31aaf612/pulsar-opentelemetry/src/main/java/org/apache/pulsar/opentelemetry/OpenTelemetryService.java#L82, while OpenTelemetry is adding others such as host name, operating system, OTel version etc.



##########
docs/reference-metrics-opentelemetry.md:
##########
@@ -402,3 +402,62 @@ The number of pending topic load operations in the broker. When it reaches thres
 The maximum number of pending topic load operations in the broker. Equal to "maxConcurrentTopicLoadRequest" defined in broker.conf.
 * Type: UpDownCounter
 * Unit: `{operation}`
+
+### Producer metrics
+
+#### pulsar.broker.producer.message.incoming.count
+The total number of messages received from this producer.
+* Type: Counter
+* Unit: `{message}`
+* Attributes:
+  * `pulsar.domain` - The domain of the topic. Can be one of:
+    * `persistent`
+    * `non-persistent`
+  * `pulsar.tenant` - The topic tenant.
+  * `pulsar.namespace` - The topic namespace.
+  * `pulsar.topic` - The topic name.
+  * `pulsar.producer.name` - The name of the producer.
+  * `pulsar.producer.id` - The ID of the producer.
+  * `pulsar.producer.access_mode` - The access mode of the producer. Can be one of:
+    * `shared`
+    * `exclusive`
+    * `wait_for_exclusive`
+    * `exclusive_with_fencing`
+
+#### pulsar.broker.producer.message.incoming.size
+The total number of messages bytes received from this producer.
+* Type: Counter
+* Unit: `By`

Review Comment:
   Yes. OpenTelemetry mandates using international units where possible. For bytes, the unit is then `By`. See https://opentelemetry.io/docs/specs/semconv/general/metrics/#instrument-units and https://unitsofmeasure.org/ucum#para-48. 



##########
docs/reference-metrics-opentelemetry.md:
##########
@@ -402,3 +402,62 @@ The number of pending topic load operations in the broker. When it reaches thres
 The maximum number of pending topic load operations in the broker. Equal to "maxConcurrentTopicLoadRequest" defined in broker.conf.
 * Type: UpDownCounter
 * Unit: `{operation}`
+
+### Producer metrics
+
+#### pulsar.broker.producer.message.incoming.count
+The total number of messages received from this producer.
+* Type: Counter
+* Unit: `{message}`
+* Attributes:
+  * `pulsar.domain` - The domain of the topic. Can be one of:
+    * `persistent`
+    * `non-persistent`
+  * `pulsar.tenant` - The topic tenant.
+  * `pulsar.namespace` - The topic namespace.
+  * `pulsar.topic` - The topic name.
+  * `pulsar.producer.name` - The name of the producer.
+  * `pulsar.producer.id` - The ID of the producer.
+  * `pulsar.producer.access_mode` - The access mode of the producer. Can be one of:
+    * `shared`
+    * `exclusive`
+    * `wait_for_exclusive`
+    * `exclusive_with_fencing`
+
+#### pulsar.broker.producer.message.incoming.size
+The total number of messages bytes received from this producer.
+* Type: Counter
+* Unit: `By`
+* Attributes:
+  * `pulsar.domain` - The domain of the topic. Can be one of:
+    * `persistent`
+    * `non-persistent`
+  * `pulsar.tenant` - The topic tenant.
+  * `pulsar.namespace` - The topic namespace.
+  * `pulsar.topic` - The topic name.
+  * `pulsar.producer.name` - The name of the producer.
+  * `pulsar.producer.id` - The ID of the producer.
+  * `pulsar.producer.access_mode` - The access mode of the producer. Can be one of:
+    * `shared`
+    * `exclusive`
+    * `wait_for_exclusive`
+    * `exclusive_with_fencing`
+
+#### pulsar.broker.producer.message.drop.count
+The total number of messages dropped from this producer.

Review Comment:
   How does this sound?
   
   ```suggestion
   The total number of messages dropped from this producer. Pending messages published to non-persistent topics can be dropped when they accumulate in excess of broker config option `maxConcurrentNonPersistentMessagePerConnection`.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [feat][doc] PIP-264: Add OpenTelemetry producer metrics reference [pulsar-site]

Posted by "heesung-sn (via GitHub)" <gi...@apache.org>.
heesung-sn commented on code in PR #911:
URL: https://github.com/apache/pulsar-site/pull/911#discussion_r1636932061


##########
docs/reference-metrics-opentelemetry.md:
##########
@@ -402,3 +402,62 @@ The number of pending topic load operations in the broker. When it reaches thres
 The maximum number of pending topic load operations in the broker. Equal to "maxConcurrentTopicLoadRequest" defined in broker.conf.
 * Type: UpDownCounter
 * Unit: `{operation}`
+
+### Producer metrics
+
+#### pulsar.broker.producer.message.incoming.count
+The total number of messages received from this producer.
+* Type: Counter
+* Unit: `{message}`
+* Attributes:
+  * `pulsar.domain` - The domain of the topic. Can be one of:
+    * `persistent`
+    * `non-persistent`
+  * `pulsar.tenant` - The topic tenant.
+  * `pulsar.namespace` - The topic namespace.
+  * `pulsar.topic` - The topic name.
+  * `pulsar.producer.name` - The name of the producer.
+  * `pulsar.producer.id` - The ID of the producer.
+  * `pulsar.producer.access_mode` - The access mode of the producer. Can be one of:
+    * `shared`
+    * `exclusive`
+    * `wait_for_exclusive`
+    * `exclusive_with_fencing`
+
+#### pulsar.broker.producer.message.incoming.size
+The total number of messages bytes received from this producer.
+* Type: Counter
+* Unit: `By`

Review Comment:
   Is `By` bytes?



##########
docs/reference-metrics-opentelemetry.md:
##########
@@ -402,3 +402,62 @@ The number of pending topic load operations in the broker. When it reaches thres
 The maximum number of pending topic load operations in the broker. Equal to "maxConcurrentTopicLoadRequest" defined in broker.conf.
 * Type: UpDownCounter
 * Unit: `{operation}`
+
+### Producer metrics
+
+#### pulsar.broker.producer.message.incoming.count
+The total number of messages received from this producer.
+* Type: Counter
+* Unit: `{message}`
+* Attributes:

Review Comment:
   Do we include the `pulsar.cluster` info too?



##########
docs/reference-metrics-opentelemetry.md:
##########
@@ -402,3 +402,62 @@ The number of pending topic load operations in the broker. When it reaches thres
 The maximum number of pending topic load operations in the broker. Equal to "maxConcurrentTopicLoadRequest" defined in broker.conf.
 * Type: UpDownCounter
 * Unit: `{operation}`
+
+### Producer metrics
+
+#### pulsar.broker.producer.message.incoming.count
+The total number of messages received from this producer.
+* Type: Counter
+* Unit: `{message}`
+* Attributes:
+  * `pulsar.domain` - The domain of the topic. Can be one of:
+    * `persistent`
+    * `non-persistent`
+  * `pulsar.tenant` - The topic tenant.
+  * `pulsar.namespace` - The topic namespace.
+  * `pulsar.topic` - The topic name.
+  * `pulsar.producer.name` - The name of the producer.
+  * `pulsar.producer.id` - The ID of the producer.
+  * `pulsar.producer.access_mode` - The access mode of the producer. Can be one of:
+    * `shared`
+    * `exclusive`
+    * `wait_for_exclusive`
+    * `exclusive_with_fencing`
+
+#### pulsar.broker.producer.message.incoming.size
+The total number of messages bytes received from this producer.
+* Type: Counter
+* Unit: `By`
+* Attributes:
+  * `pulsar.domain` - The domain of the topic. Can be one of:
+    * `persistent`
+    * `non-persistent`
+  * `pulsar.tenant` - The topic tenant.
+  * `pulsar.namespace` - The topic namespace.
+  * `pulsar.topic` - The topic name.
+  * `pulsar.producer.name` - The name of the producer.
+  * `pulsar.producer.id` - The ID of the producer.
+  * `pulsar.producer.access_mode` - The access mode of the producer. Can be one of:
+    * `shared`
+    * `exclusive`
+    * `wait_for_exclusive`
+    * `exclusive_with_fencing`
+
+#### pulsar.broker.producer.message.drop.count
+The total number of messages dropped from this producer.

Review Comment:
   could we clarify "dropped messages" more or put a reference link? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org