You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2019/10/04 21:55:31 UTC
[GitHub] [pulsar] digi691 opened a new issue #5318: Pulsar 2.4.1 - Broker
stops server Admin API at random
digi691 opened a new issue #5318: Pulsar 2.4.1 - Broker stops server Admin API at random
URL: https://github.com/apache/pulsar/issues/5318
**Describe the bug**
The broker stops serving the Admin API. When it stops serving the Admin API, there is a corresponding WARN message from org.eclipse.jetty.util.thread.strategy.EatWhatYouKill. Right now the issue is random but seems to be happening every few hours. Restarting the Broker is the only way to recover.
**To Reproduce**
Steps to reproduce the behavior:
The cluster has two HAProxy load balancing to two Pulsar Proxies. Behind the Proxies there are 3 Pulsar Brokers, and 4 Pulsar Bookies. The behavior is random, though usually happens every 2-3 hours to a broker in the cluster. If I let the problem go long enough, it will affect all 3 brokers.
**Expected behavior**
Admi API is served without disruptions.
**Screenshots**
The Log message below that corresponds with the time the broker stops serving HTTP(S) requests.
```
2019-10-04 13:58:12.292 [pulsar-web-29-5] WARN org.eclipse.jetty.util.thread.strategy.EatWhatYouKill -
java.nio.channels.CancelledKeyException: null
at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:71) ~[?:?]
at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:130) ~[?:?]
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.processSelected(ManagedSelector.java:473) ~[org.eclipse.jetty-jetty-io-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:351) ~[org.eclipse.jetty-jetty-io-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) [org.eclipse.jetty-jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) [org.eclipse.jetty-jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) [org.eclipse.jetty-jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) [org.eclipse.jetty-jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) [org.eclipse.jetty-jetty-util-9.4.12.v20180830.jar:9.4.12.v20180830]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final]
at java.lang.Thread.run(Thread.java:834) [?:?]
```
**Desktop (please complete the following information):**
- OS: [Ubuntu]
**Additional context**
None
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services