[GitHub] [pulsar] wangjialing218 opened a new issue #11695: Stop one broker may cause another broker shutdown when enable function woker in broker

[GitBox <gi...@apache.org>]

wangjialing218 opened a new issue #11695:
URL: https://github.com/apache/pulsar/issues/11695


   **To Reproduce**
   1. Start pulsar cluster with 3 brokers and `functionsWorkerEnabled=true`
   2. Stop the broker which is the leader of function worker and the owner of function worker coordinate topic.
   3. Another broker will become leader and then shutdown due to fail to get coordinate topic.
   
   **Expected behavior**
   Should not trigger shutdown when fail to get coordinate topic.
   
   **Additional context**
   When stop the broker which is the leader of function woker and the owner of coordinate topic. Another broker will become leader but coordinate topic did not shift to another broker in time.
   There is a schedule task runing only on leader and will get follow error:
   
   > ERROR org.apache.pulsar.functions.worker.ErrorNotifierShutdownServiceImpl - Encountered fatal error. Shutting down.
   > java.lang.RuntimeException: org.apache.pulsar.client.admin.PulsarAdminException: java.util.concurrent.CompletionException: org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException: Could not complete the operation. Number of retries has been exhausted. Failed reason: Connection refused: /xxx.xxx.xxx.xxx:8080
   > 	at org.apache.pulsar.functions.worker.MembershipManager.getCurrentMembership(MembershipManager.java:78) 
   
   `getCurrentMembership` will throw a `RuntimeException` when fail to get coordinate topic. And because we are using `ErrorNotifierShutdownServiceImpl` the broker will shutdown when get exception.
   
   There may be two parts need to be improved:
   1. Add error handle when get coordinate topic.
   2. Do not use `ErrorNotifierShutdownServiceImpl`, it is not suitable to shutdown broker when function worker get any exception.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] codelipenghui commented on issue #11695: Stop one broker may cause another broker shutdown when enable function woker in broker

[GitBox <gi...@apache.org>]

codelipenghui commented on issue #11695:
URL: https://github.com/apache/pulsar/issues/11695#issuecomment-1058887148


   The issue had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org