You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/11/15 05:32:01 UTC
[GitHub] [pulsar] MarvinCai opened a new pull request #8567: Add key-shared consumer range to internal topic stats.
MarvinCai opened a new pull request #8567:
URL: https://github.com/apache/pulsar/pull/8567
Fixes #5752
Master Issue: #4077
### Motivation
Expose consumer range information in topic internal stats.
### Modifications
### Verifying this change
This change added tests and can be verified as follows:
- Add unit test to verify output expected range info.
### Does this pull request potentially affect one of the following parts:
*If `yes` was chosen, please highlight the changes*
- Dependencies (does it add or upgrade a dependency): no
- The public API: no
- The schema: no
- The default values of configurations: no
- The wire protocol: no
- The rest endpoints: no
- The admin cli options: no
- Anything that affects deployment: no
### Documentation
- Does this pull request introduce a new feature? no
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] MarvinCai commented on pull request #8567: Add key-shared consumer range to internal topic stats.
Posted by GitBox <gi...@apache.org>.
MarvinCai commented on pull request #8567:
URL: https://github.com/apache/pulsar/pull/8567#issuecomment-727606320
/pulsarbot run-failure-checks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] MarvinCai commented on pull request #8567: Add key-shared consumer range to internal topic stats.
Posted by GitBox <gi...@apache.org>.
MarvinCai commented on pull request #8567:
URL: https://github.com/apache/pulsar/pull/8567#issuecomment-727772828
Update to put this info in ConsumerStats and it's now part of topic-stats, example output:
```
{
"msgRateIn" : 0.0,
"msgThroughputIn" : 0.0,
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"bytesInCounter" : 0,
"msgInCounter" : 0,
"bytesOutCounter" : 0,
"msgOutCounter" : 0,
"averageMsgSize" : 0.0,
"msgChunkPublished" : false,
"storageSize" : 0,
"backlogSize" : 0,
"publishers" : [ ],
"subscriptions" : {
"zxcc-sub" : {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"bytesOutCounter" : 0,
"msgOutCounter" : 0,
"msgRateRedeliver" : 0.0,
"chuckedMessageRate" : 0,
"msgBacklog" : 0,
"msgBacklogNoDelayed" : 0,
"blockedSubscriptionOnUnackedMsgs" : false,
"msgDelayed" : 0,
"unackedMessages" : 0,
"type" : "Key_Shared",
"msgRateExpired" : 0.0,
"lastExpireTimestamp" : 0,
"lastConsumedFlowTimestamp" : 1605508943123,
"lastConsumedTimestamp" : 0,
"lastAckedTimestamp" : 0,
"consumers" : [ {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"bytesOutCounter" : 0,
"msgOutCounter" : 0,
"msgRateRedeliver" : 0.0,
"chuckedMessageRate" : 0.0,
"consumerName" : "97a6d",
"availablePermits" : 1000,
"unackedMessages" : 0,
"avgMessagesPerEntry" : 1000,
"blockedConsumerOnUnackedMsgs" : false,
"readPositionWhenJoining" : "179:0",
"lastAckedTimestamp" : 0,
"lastConsumedTimestamp" : 0,
"keyHashRange" : [ "32769--65536" ],
"metadata" : { },
"connectedSince" : "2020-11-15T22:42:18.625-08:00",
"clientVersion" : "2.7.0-SNAPSHOT",
"address" : "/127.0.0.1:58473"
}, {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"bytesOutCounter" : 0,
"msgOutCounter" : 0,
"msgRateRedeliver" : 0.0,
"chuckedMessageRate" : 0.0,
"consumerName" : "eb10d",
"availablePermits" : 1000,
"unackedMessages" : 0,
"avgMessagesPerEntry" : 1000,
"blockedConsumerOnUnackedMsgs" : false,
"readPositionWhenJoining" : "179:0",
"lastAckedTimestamp" : 0,
"lastConsumedTimestamp" : 0,
"keyHashRange" : [ "0--32768" ],
"metadata" : { },
"connectedSince" : "2020-11-15T22:42:23.118-08:00",
"clientVersion" : "2.7.0-SNAPSHOT",
"address" : "/127.0.0.1:58496"
} ],
"isDurable" : true,
"isReplicated" : false,
"consumersAfterMarkDeletePosition" : { }
},
"zxc-sub" : {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"bytesOutCounter" : 0,
"msgOutCounter" : 0,
"msgRateRedeliver" : 0.0,
"chuckedMessageRate" : 0,
"msgBacklog" : 0,
"msgBacklogNoDelayed" : 0,
"blockedSubscriptionOnUnackedMsgs" : false,
"msgDelayed" : 0,
"unackedMessages" : 0,
"msgRateExpired" : 0.0,
"lastExpireTimestamp" : 0,
"lastConsumedFlowTimestamp" : 0,
"lastConsumedTimestamp" : 0,
"lastAckedTimestamp" : 0,
"consumers" : [ ],
"isDurable" : true,
"isReplicated" : false,
"consumersAfterMarkDeletePosition" : { }
}
},
"replication" : { },
"deduplicationStatus" : "Disabled"
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] MarvinCai commented on pull request #8567: Add key-shared consumer range to internal topic stats.
Posted by GitBox <gi...@apache.org>.
MarvinCai commented on pull request #8567:
URL: https://github.com/apache/pulsar/pull/8567#issuecomment-727612850
@codelipenghui can you please help take a look
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on a change in pull request #8567: Add key-shared consumer range to internal topic stats.
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on a change in pull request #8567:
URL: https://github.com/apache/pulsar/pull/8567#discussion_r523964261
##########
File path: pulsar-common/src/main/java/org/apache/pulsar/common/policies/data/ConsumerStats.java
##########
@@ -77,6 +78,9 @@
public long lastAckedTimestamp;
public long lastConsumedTimestamp;
+ /** Hash range assigned to this consumer if is Key_Shared sub mode. **/
+ public List<String> keyHashRange;
Review comment:
```suggestion
public List<String> keyHashRanges;
```
##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStickyKeyDispatcherMultipleConsumers.java
##########
@@ -369,6 +369,10 @@ public SubType getType() {
return recentlyJoinedConsumers;
}
+ public Map<String, List<String>> getConsumerRange() {
Review comment:
```suggestion
public Map<String, List<String>> getConsumerKeyHashRanges() {
```
##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/service/StickyKeyConsumerSelector.java
##########
@@ -43,4 +46,10 @@
* @return consumer
*/
Consumer select(byte[] stickyKey);
+
+ /**
+ * Get range handled by each consumer
+ * @return A map where key is a consumer name and value is list of range it receiving message for.
+ */
+ Map<String, List<String>> getConsumerRange();
Review comment:
```suggestion
Map<String, List<String>> getConsumerKeyHashRanges();
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] MarvinCai commented on pull request #8567: Add key-shared consumer range to internal topic stats.
Posted by GitBox <gi...@apache.org>.
MarvinCai commented on pull request #8567:
URL: https://github.com/apache/pulsar/pull/8567#issuecomment-728130083
agreed, it's more like interval and we should clearly specify if it's open or closed interval, updated the string format.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] MarvinCai commented on pull request #8567: Add key-shared consumer range to internal topic stats.
Posted by GitBox <gi...@apache.org>.
MarvinCai commented on pull request #8567:
URL: https://github.com/apache/pulsar/pull/8567#issuecomment-728274245
/pulsarbot run-failure-checks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] MarvinCai edited a comment on pull request #8567: Add key-shared consumer range to internal topic stats.
Posted by GitBox <gi...@apache.org>.
MarvinCai edited a comment on pull request #8567:
URL: https://github.com/apache/pulsar/pull/8567#issuecomment-728130083
make sense, the key range is more like interval and it's better we clearly specify if it's open or closed interval, updated the string format.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] MarvinCai commented on a change in pull request #8567: Add key-shared consumer range to internal topic stats.
Posted by GitBox <gi...@apache.org>.
MarvinCai commented on a change in pull request #8567:
URL: https://github.com/apache/pulsar/pull/8567#discussion_r524357803
##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStickyKeyDispatcherMultipleConsumers.java
##########
@@ -369,6 +369,10 @@ public SubType getType() {
return recentlyJoinedConsumers;
}
+ public Map<String, List<String>> getConsumerRange() {
Review comment:
updated
##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/service/StickyKeyConsumerSelector.java
##########
@@ -43,4 +46,10 @@
* @return consumer
*/
Consumer select(byte[] stickyKey);
+
+ /**
+ * Get range handled by each consumer
+ * @return A map where key is a consumer name and value is list of range it receiving message for.
+ */
+ Map<String, List<String>> getConsumerRange();
Review comment:
updated
##########
File path: pulsar-common/src/main/java/org/apache/pulsar/common/policies/data/ConsumerStats.java
##########
@@ -77,6 +78,9 @@
public long lastAckedTimestamp;
public long lastConsumedTimestamp;
+ /** Hash range assigned to this consumer if is Key_Shared sub mode. **/
+ public List<String> keyHashRange;
Review comment:
updated
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #8567: Add key-shared consumer range to internal topic stats.
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on pull request #8567:
URL: https://github.com/apache/pulsar/pull/8567#issuecomment-727668814
@MarvinCai I think it's better to add the consumer-related stats into the `topics stats`. The `topics stats-internal` expose the ledger and cursor stats, the `topics stats` expose the producer, subscription, and consumer stats. So `topics stats` seems more reasonable.
A consumer might be assigned to multiple `hash range`. So I think you can add a `keyHashRanges` in the consumer stats directly,
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] MarvinCai edited a comment on pull request #8567: Add key-shared consumer range to internal topic stats.
Posted by GitBox <gi...@apache.org>.
MarvinCai edited a comment on pull request #8567:
URL: https://github.com/apache/pulsar/pull/8567#issuecomment-728130083
agreed, it's more like interval and it's better we clearly specify if it's open or closed interval, updated the string format.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] MarvinCai commented on pull request #8567: Add key-shared consumer range to internal topic stats.
Posted by GitBox <gi...@apache.org>.
MarvinCai commented on pull request #8567:
URL: https://github.com/apache/pulsar/pull/8567#issuecomment-728249249
/pulsarbot run-failure-checks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui merged pull request #8567: Add key-shared consumer range to internal topic stats.
Posted by GitBox <gi...@apache.org>.
codelipenghui merged pull request #8567:
URL: https://github.com/apache/pulsar/pull/8567
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org