You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/03/08 05:51:28 UTC
[GitHub] [pulsar] massakam opened a new pull request #9835: [broker] Continue graceful shutdown even if web service closing fails
massakam opened a new pull request #9835:
URL: https://github.com/apache/pulsar/pull/9835
### Motivation
Occasionally, the web service closing times out when shutting down the broker server. The following is the error log at that time:
```
13:38:20.200 [Thread-2] ERROR o.a.p.b.MessagingServiceShutdownHook - Failed to perform graceful shutdown, Exiting anyway
java.util.concurrent.ExecutionException: org.apache.pulsar.broker.PulsarServerException: org.apache.pulsar.broker.PulsarServerException: java.util.concurrent.TimeoutException
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
at org.apache.pulsar.broker.MessagingServiceShutdownHook.run(MessagingServiceShutdownHook.java:69)
at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.shutdown(PulsarBrokerStarter.java:294)
at org.apache.pulsar.PulsarBrokerStarter.lambda$main$1(PulsarBrokerStarter.java:322)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.pulsar.broker.PulsarServerException: org.apache.pulsar.broker.PulsarServerException: java.util.concurrent.TimeoutException
at org.apache.pulsar.broker.PulsarService.close(PulsarService.java:316)
at org.apache.pulsar.broker.MessagingServiceShutdownHook.lambda$run$0(MessagingServiceShutdownHook.java:62)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
Caused by: org.apache.pulsar.broker.PulsarServerException: java.util.concurrent.TimeoutException
at org.apache.pulsar.broker.web.WebService.close(WebService.java:203)
at org.apache.pulsar.broker.PulsarService.close(PulsarService.java:229)
... 5 common frames omitted
Caused by: java.util.concurrent.TimeoutException: null
at org.eclipse.jetty.util.FutureCallback.get(FutureCallback.java:130)
at org.eclipse.jetty.util.FutureCallback.get(FutureCallback.java:30)
at org.eclipse.jetty.server.handler.AbstractHandlerContainer.doShutdown(AbstractHandlerContainer.java:175)
at org.eclipse.jetty.server.Server.doStop(Server.java:447)
at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:93)
at org.apache.pulsar.broker.web.WebService.close(WebService.java:199)
... 6 common frames omitted
```
If this happens, the JVM process will be killed without closing the broker service, so topics owned by this broker will be moved to other brokers without their cursor information being persisted to the cursor ledgers. As a result, those cursors rewind to the old position and the duplicate messages are delivered to the consumers.
### Modifications
Even if an exception is thrown when closing the web service (i.e. Jetty server), catche the exception and continue the graceful shutdown process. As far as I can see the Jetty source code, it seems that the stop process is completed even if an exception occurs while the Jetty server is stopped.
https://github.com/eclipse/jetty.project/blob/jetty-9.4.33.v20201020/jetty-server/src/main/java/org/eclipse/jetty/server/Server.java#L427-L485
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] merlimat merged pull request #9835: [broker] Continue graceful shutdown even if web service closing fails
Posted by GitBox <gi...@apache.org>.
merlimat merged pull request #9835:
URL: https://github.com/apache/pulsar/pull/9835
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org