You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2023/01/11 08:08:28 UTC

[GitHub] [pulsar] poorbarcode opened a new issue, #19187: PIP-240: Unloading subscribers

poorbarcode opened a new issue, #19187:
URL: https://github.com/apache/pulsar/issues/19187

   ### Motivation
   
   We sometimes try to unload the topic to resolve some consumption-stop issues. But the unloading topic will also impact the producer side.
   
   ### Goal
   
   Providing a new API to perform unmounting of the subscription dimension triggers reconnection of all consumers on that subscription (from the client's perspective, this connection may be shared by consumers, producers, and transactions) and reconnection is guaranteed by the client. The API will be used in these ways:
   - unload special subscription of one topic(or partitioned topic)
   - unload all subscriptions of one topic(or partitioned topic)
   - unload subscriptions of one topic(or partitioned topic) by regular expression
     - If a reader's subscription name is not set, a random subscription name prefixed with 'multiTopicsReader-' or 'reader-' will be used, and users can uninstall these subscriptions using regular expressions. 
   
   In addition to triggering consumer disconnection, Unloading Subscribers will restart the Dispatcher, which resets the redeliver message queue and delayed message queue in the Broker's memory, which can help resolve issues caused by an abnormal dispatcher state. However, the execution flow of Unloading Subscribers does not include a restart of the Managed Cursor related to this dispatcher; if there is a problem with the cursor, we can only rely on the unload topic to solve it.
   
   #### These scenarios are not supported
   - Functions `message-dedup`, `geo-replication,` and `shadow-topic` also read messages from the topic, but the feature Unloading subscribers will not support triggering restarts of these three functions( because the cursor is used directly to read the data, not the consumer or reader ).
   - The Compression task(subscription name is `__compaction`) and Recovery of the Transaction Buffer also use a reader to read data, but Unloading Subscribers does not trigger a reconnection because both tasks create a new reader each time they start.
   
   #### Special system topic supports
   The system topic `__change_events` is used to support topic-level policies, Unloading Subscribers will support the reconnection of readers on this topic.
   
   ### API Changes
   
   v2.PersistentTopics
   ```java
   /**
    * Unload {sub} on {topic}.
    * @param subName It can be a subscription name or a regular expression. "**" represents all the subscriptions under the topic.
    *  It responds with error code 412 When the user tries to unload these subscriptions: `__compaction`, `dedup`, `pulsar.repl.*`.
   **/
   @Path("/{tenant}/{namespace}/{topic}/{subName}/unload")
   public void unloadTopic(...)
   ```
   
   ### Implementation
   
   -
   
   ### Alternatives
   
   _No response_
   
   ### Anything else?
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] github-actions[bot] commented on issue #19187: PIP-240: A new API to unload subscriptions

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #19187:
URL: https://github.com/apache/pulsar/issues/19187#issuecomment-1433993751

   The issue had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] eolivelli commented on issue #19187: PIP-240: A new API to unload subscriptions

Posted by GitBox <gi...@apache.org>.
eolivelli commented on issue #19187:
URL: https://github.com/apache/pulsar/issues/19187#issuecomment-1379146832

   In my experience if a subscription is stuck, the problems are:
   - the client has some problems - 99.9%
   - there is a bug in Pulsar - 0.1%
   
   Unloading a topic is an operation that triggers some reset of the state on both the broker and the clients and this usually TEMPORARY unblocks the subscription.
   
   If there is a problem we should spend time on investigating the problem and not in adding this kind of tools.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] dave2wave commented on issue #19187: PIP-240: A new API to unload subscriptions

Posted by GitBox <gi...@apache.org>.
dave2wave commented on issue #19187:
URL: https://github.com/apache/pulsar/issues/19187#issuecomment-1379204318

   I think that the terminology of `unload` is wrong. These are `reset` operations.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] PIP-240: A new API to unload subscriptions [pulsar]

Posted by "dao-jun (via GitHub)" <gi...@apache.org>.
dao-jun closed issue #19187: PIP-240: A new API to unload subscriptions
URL: https://github.com/apache/pulsar/issues/19187


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] zbentley commented on issue #19187: PIP-240: A new API to reset subscriptions

Posted by GitBox <gi...@apache.org>.
zbentley commented on issue #19187:
URL: https://github.com/apache/pulsar/issues/19187#issuecomment-1380967371

   As much as I also like `unload`ing topics to resolve mysterious issues, I'd rather pulsar not add additional "band-aid/manual reset" command functionality to work around either misbehaving clients or broker state-corrupting bugs. I think doing that has a short-term benefit of providing a tool for expert operators to use to resolve issues, but has a long-term cost of making brand new adopters of pulsar find it less reliable than they want it to be.
   
   I'm not saying "fix the bugs _instead_" (though we should also fix things!), but rather that building more support code for stuff like this may come at a time/complexity cost with actually fixing the issues that would make a "reset" necessary.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] poorbarcode commented on issue #19187: PIP-240: A new API to reset subscriptions

Posted by GitBox <gi...@apache.org>.
poorbarcode commented on issue #19187:
URL: https://github.com/apache/pulsar/issues/19187#issuecomment-1379929522

   Hi @eolivelli 
   
   > If there is a problem we should spend time on investigating the problem and not in adding this kind of tools.
   
   You are right. When a user encounters a problem, it often takes a while actually to solve the root cause. It is important to provide a tool to recover services quickly, and cmd 'unload topic' is often used to solve problems temporarily. 
   
   But now a topic can be used by multiple teams, such as Business-team A to produce messages, Business-team B and C to consume messages (each using a different subscription name), and Data-team D to read messages using reader api (using a random subscription name). When there is a consumption problem in subscription B, if we do unload the topic, this will affect teams A, B, C, D, and when we provide a new API to reset subscribers, the impact can be controlled only to affect team B.
   
   So I think a new API is necessary。
   
   Thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [pulsar] poorbarcode commented on issue #19187: PIP-240: A new API to reset subscriptions

Posted by GitBox <gi...@apache.org>.
poorbarcode commented on issue #19187:
URL: https://github.com/apache/pulsar/issues/19187#issuecomment-1379929956

   Hi @dave2wave 
   
   > I think that the terminology of unload is wrong. These are reset operations.
   
   already change `unload` to `reset`
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org