[GitHub] [pulsar] xche opened a new issue #8575: When a task is stopped or deleted, it will cause all other tasks to stop consuming data

[GitBox <gi...@apache.org>]

xche opened a new issue #8575:
URL: https://github.com/apache/pulsar/issues/8575


   **Describe the bug**
   Runtime ： Thread Runtime
   broker number ：3
   bookie number ：6 
   functionWorker number ： 6
   task number: 423
   
   **When I call the following interfaces, all other tasks will stop consuming data.**
   https://pulsar.apache.org/admin/v3/sinks/{tenant}/{namespace}/{sinkName}/stop
   **When all tasks stop consuming data, we can see the following stack information.**
   ![image](https://user-images.githubusercontent.com/1579796/99225163-93498480-2822-11eb-9ac1-94380ba396e7.png)
   ![image](https://user-images.githubusercontent.com/1579796/99225194-a197a080-2822-11eb-8f0b-d93876d64b89.png)
   **And the topic stats:**
   ![image](https://user-images.githubusercontent.com/1579796/99225740-811c1600-2823-11eb-9250-29e10f2d8a41.png)
   
   
   The reason for this problem is that there is only one PulsarClient in a functionWorker instance, and the number of externalExecutorProvider thread pools in PulsarClient is 1, so all consumers share the same listener executor. Therefore, when one or more tasks stoped, the listener executor will be blocked, and finally other tasks will stop consuming data.
   
   
   I think there should be at least two solutions:
   1. Add "numListenerThreads" configuration in WorkerConfig.
   2. Remove the message listener in PulsarSource, and add a thread in PulsarSource to execute the consumer's receive method 
    to take message. 
   
   There is a problem with scheme 1: when you stop or delete a task, you will see the following in the stack
   ![image](https://user-images.githubusercontent.com/1579796/99225163-93498480-2822-11eb-9ac1-94380ba396e7.png)
   
   
   
   
   How do you like it? Or are there any other, more graceful, solutions to this problem?
   
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org