You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/03/04 01:15:10 UTC
[GitHub] [pulsar] rdhabalia opened a new pull request #9789: [pulsar-broker] Allow broker to discover and unblock stuck subscription
rdhabalia opened a new pull request #9789:
URL: https://github.com/apache/pulsar/pull/9789
### Motivation
We have been frequently seeing issue where subscription gets stuck on different topics and broker is not dispatching messages though consumer has available-permits and no pending reads (example #9788). It can happen due to regression bug or unknown issue when expiry runs.. one of the workarounds is manually unload the topic and reload it which is not feasible if this happens frequently to many topics. Or broker should have the capability to discover such stuck subscriptions and unblock them.
Below example shows that:
subscription has available-permit>0, there is no pending reads, cursor's read-position is not moving forward and that builds the backlog until we unload the topic. It happens frequently due to unknown reason:
```
STATS-INTERNAL:
"sub1" : {
"markDeletePosition" : "11111111:15520",
"readPosition" : "11111111:15521",
"waitingReadOp" : false,
"pendingReadOps" : 0,
"messagesConsumedCounter" : 115521,
"cursorLedger" : 585099247,
"cursorLedgerLastEntry" : 597,
"individuallyDeletedMessages" : "[]",
"lastLedgerSwitchTimestamp" : "2021-02-25T19:55:50.357Z",
"state" : "Open",
"numberOfEntriesSinceFirstNotAckedMessage" : 1,
"totalNonContiguousDeletedMessagesRange" : 0,
STATS:
"sub1" : {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"msgRateRedeliver" : 0.0,
"msgBacklog" : 30350,
"blockedSubscriptionOnUnackedMsgs" : false,
"msgDelayed" : 0,
"unackedMessages" : 0,
"type" : "Shared",
"msgRateExpired" : 0.0,
"consumers" : [ {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"msgRateRedeliver" : 0.0,
"consumerName" : "C1",
"availablePermits" : 723,
"unackedMessages" : 0,
"blockedConsumerOnUnackedMsgs" : false,
"metadata" : { },
"connectedSince" : "2021-02-25T19:55:50.358285Z",
```
### Modification
Add capability in broker to periodically check if subscription is stuck and unblock it if needed. This check is controlled by flag and for initial release it can be disabled by default (and we can enable by default in later release)
### Result
It helps broker to handle stuck subscription and logs the message for later debugging.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #9789: [pulsar-broker] Allow broker to discover and unblock stuck subscription
Posted by GitBox <gi...@apache.org>.
lhotari commented on pull request #9789:
URL: https://github.com/apache/pulsar/pull/9789#issuecomment-791499309
> We have been frequently seeing issue where subscription gets stuck on different topics and broker is not dispatching messages though consumer has available-permits and no pending reads (example #9788). It can happen due to regression bug or unknown issue when expiry runs.
This could be some sort of locked up state which is prevented by @merlimat 's change #9787 . @rdhabalia do you have some environment where you could check if #9787 helps to fix the issue you are seeing?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] rdhabalia commented on a change in pull request #9789: [pulsar-broker] Allow broker to discover and unblock stuck subscription
Posted by GitBox <gi...@apache.org>.
rdhabalia commented on a change in pull request #9789:
URL: https://github.com/apache/pulsar/pull/9789#discussion_r588710660
##########
File path: conf/broker.conf
##########
@@ -275,6 +275,9 @@ maxUnackedMessagesPerBroker=0
# limit/2 messages
maxUnackedMessagesPerSubscriptionOnBrokerBlocked=0.16
+# Broker periodically checks if subscription is stuck and unblock if flag is enabled. (Default is disabled)
+unblockStuckSubscriptionEnabled=false
Review comment:
this check depends on the rate and this check is performed in the same stats-update task so, this feature doesn't require additional configuration.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] devinbost commented on pull request #9789: [pulsar-broker] Allow broker to discover and unblock stuck subscription
Posted by GitBox <gi...@apache.org>.
devinbost commented on pull request #9789:
URL: https://github.com/apache/pulsar/pull/9789#issuecomment-800441782
Also, if you restarted the broker that owned the topic, would that resolve the issue?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] devinbost edited a comment on pull request #9789: [pulsar-broker] Allow broker to discover and unblock stuck subscription
Posted by GitBox <gi...@apache.org>.
devinbost edited a comment on pull request #9789:
URL: https://github.com/apache/pulsar/pull/9789#issuecomment-800366588
It seems like this is somewhat of a bandaid. How do we find the root cause of why subscriptions are getting stuck? This seems like it might be related to https://github.com/apache/pulsar/issues/6054
@rdhabalia
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on a change in pull request #9789: [pulsar-broker] Allow broker to discover and unblock stuck subscription
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on a change in pull request #9789:
URL: https://github.com/apache/pulsar/pull/9789#discussion_r587924752
##########
File path: conf/broker.conf
##########
@@ -275,6 +275,9 @@ maxUnackedMessagesPerBroker=0
# limit/2 messages
maxUnackedMessagesPerSubscriptionOnBrokerBlocked=0.16
+# Broker periodically checks if subscription is stuck and unblock if flag is enabled. (Default is disabled)
+unblockStuckSubscriptionEnabled=false
Review comment:
From the description of the flag, the broker will periodically check if stuck and unblock, Should we inform users what is the default frequency and how to change the frequency.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] rdhabalia commented on pull request #9789: [pulsar-broker] Allow broker to discover and unblock stuck subscription
Posted by GitBox <gi...@apache.org>.
rdhabalia commented on pull request #9789:
URL: https://github.com/apache/pulsar/pull/9789#issuecomment-791808744
/pulsarbot run-failure-checks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] merlimat commented on a change in pull request #9789: [pulsar-broker] Allow broker to discover and unblock stuck subscription
Posted by GitBox <gi...@apache.org>.
merlimat commented on a change in pull request #9789:
URL: https://github.com/apache/pulsar/pull/9789#discussion_r587679411
##########
File path: managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/PositionImpl.java
##########
@@ -120,11 +120,13 @@ public int hashCode() {
@Override
public boolean equals(Object obj) {
+ if (obj == null) {
Review comment:
The null scenario is already covered by the `instanceof` check
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] devinbost commented on pull request #9789: [pulsar-broker] Allow broker to discover and unblock stuck subscription
Posted by GitBox <gi...@apache.org>.
devinbost commented on pull request #9789:
URL: https://github.com/apache/pulsar/pull/9789#issuecomment-816922986
I realized that this won't help when negative permits are occurring... So, there's still more work to do to unblock stuck subscriptions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] rdhabalia commented on pull request #9789: [pulsar-broker] Allow broker to discover and unblock stuck subscription
Posted by GitBox <gi...@apache.org>.
rdhabalia commented on pull request #9789:
URL: https://github.com/apache/pulsar/pull/9789#issuecomment-791726237
@lhotari this issue is a more functional issue (mostly related expiry-check and regression bug which we saw in past with key-shared sub) and it doesn't happen due to deadlock (validated thread-dump) or slower processing because sub gets stuck until one manually unload the topic. so, #9787 won't exactly address this issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] devinbost edited a comment on pull request #9789: [pulsar-broker] Allow broker to discover and unblock stuck subscription
Posted by GitBox <gi...@apache.org>.
devinbost edited a comment on pull request #9789:
URL: https://github.com/apache/pulsar/pull/9789#issuecomment-800441782
Also, if you restarted the broker that owned the topic, would that (temporarily) resolve the issue?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] devinbost removed a comment on pull request #9789: [pulsar-broker] Allow broker to discover and unblock stuck subscription
Posted by GitBox <gi...@apache.org>.
devinbost removed a comment on pull request #9789:
URL: https://github.com/apache/pulsar/pull/9789#issuecomment-800441782
Also, if you restarted the broker that owned the topic, would that (temporarily) resolve the issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] devinbost commented on pull request #9789: [pulsar-broker] Allow broker to discover and unblock stuck subscription
Posted by GitBox <gi...@apache.org>.
devinbost commented on pull request #9789:
URL: https://github.com/apache/pulsar/pull/9789#issuecomment-815398805
@rdhabalia Thanks for doing this work. It looks like it's going to help a lot with cases where the subscriptions stop receiving messages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] devinbost commented on pull request #9789: [pulsar-broker] Allow broker to discover and unblock stuck subscription
Posted by GitBox <gi...@apache.org>.
devinbost commented on pull request #9789:
URL: https://github.com/apache/pulsar/pull/9789#issuecomment-800366588
It seems like this is somewhat of a bandaid. Wouldn't it be better for us to find the root cause of why subscriptions are getting stuck? This seems like it might be related to https://github.com/apache/pulsar/issues/6054
@rdhabalia
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui merged pull request #9789: [pulsar-broker] Allow broker to discover and unblock stuck subscription
Posted by GitBox <gi...@apache.org>.
codelipenghui merged pull request #9789:
URL: https://github.com/apache/pulsar/pull/9789
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org