You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/07/08 03:23:47 UTC
[GitHub] [pulsar] aloyszhang opened a new issue #7476: Discovery service can't find active broker
aloyszhang opened a new issue #7476:
URL: https://github.com/apache/pulsar/issues/7476
**Describe the bug**
Discovery service can't find active broker when all brokers are running well.
`21:05:33.529 [pulsar-discovery-io-2-1] WARN org.apache.pulsar.discovery.service.ServerConnection - [/100.76.40.78:34900] Failed to get next active broker No active broker is available
org.apache.pulsar.broker.PulsarServerException: No active broker is available
at org.apache.pulsar.discovery.service.BrokerDiscoveryProvider.nextBroker(BrokerDiscoveryProvider.java:95) ~[org.apache.pulsar-pulsar-discovery-service-2.6.0-SNAPSHOT.jar:2.6.0-SNAPSHOT]
at org.apache.pulsar.discovery.service.ServerConnection.sendLookupResponse(ServerConnection.java:136) [org.apache.pulsar-pulsar-discovery-service-2.6.0-SNAPSHOT.jar:2.6.0-SNAPSHOT]
at org.apache.pulsar.discovery.service.ServerConnection.handleLookup(ServerConnection.java:127) [org.apache.pulsar-pulsar-discovery-service-2.6.0-SNAPSHOT.jar:2.6.0-SNAPSHOT]
at org.apache.pulsar.common.protocol.PulsarDecoder.channelRead(PulsarDecoder.java:126) [org.apache.pulsar-pulsar-common-2.6.0-SNAPSHOT.jar:2.6.0-SNAPSHOT]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) [io.netty-netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [io.netty-netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) [io.netty-netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321) [io.netty-netty-codec-4.1.45.Final.jar:4.1.45.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295) [io.netty-netty-codec-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) [io.netty-netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [io.netty-netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) [io.netty-netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [io.netty-netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) [io.netty-netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [io.netty-netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [io.netty-netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) [io.netty-netty-transport-native-epoll-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) [io.netty-netty-transport-native-epoll-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) [io.netty-netty-transport-native-epoll-4.1.45.Final.jar:4.1.45.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [io.netty-netty-common-4.1.45.Final.jar:4.1.45.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty-netty-common-4.1.45.Final.jar:4.1.45.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.45.Final.jar:4.1.45.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]`
**To Reproduce**
After restart the discovery service , wo can't reproduce it
**Expected behavior**
Discovery service should be work well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] aloyszhang edited a comment on issue #7476: Discovery service can't find active broker
Posted by GitBox <gi...@apache.org>.
aloyszhang edited a comment on issue #7476:
URL: https://github.com/apache/pulsar/issues/7476#issuecomment-668934910
The reason we start a separate discovery service is we have number of brokers and not want to expose all of them to outside.
Since not found the exactlly reason cause this issue. We bring a compromise solution， try get active brokers from zk one more time if `availableBrokers` in `localZkCache` is empty. If zk does have brokers information, then return. @sijie
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] aloyszhang commented on issue #7476: Discovery service can't find active broker
Posted by GitBox <gi...@apache.org>.
aloyszhang commented on issue #7476:
URL: https://github.com/apache/pulsar/issues/7476#issuecomment-655260885
Addtionally, ZooKeeper has no error.
And we dump the discovery process, found that `BrokerDiscoveryProvider` does not has availableBrokers in its `ZookeeperCacheLoader` but `DiscoveryServiceServlet` has the information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] aloyszhang edited a comment on issue #7476: Discovery service can't find active broker
Posted by GitBox <gi...@apache.org>.
aloyszhang edited a comment on issue #7476:
URL: https://github.com/apache/pulsar/issues/7476#issuecomment-656712253
@sijie, Thanks for you reply.
This problem has happened several times in our pulsar cluster. At the first time, I thought it was caused by session expires, but no SessionExpired related log can be found.
```java
localZkConnectionSvc.start(new ZookeeperSessionExpiredHandler() {
@Override
public void onSessionExpired() {
log.error("Shutting down ZK sessions: {}", -1);
}
@Override
public void setWatcher(ZooKeeperSessionWatcher watcher) {
}
});
this.localZkCache = new LocalZooKeeperCache(localZkConnectionSvc.getLocalZooKeeper(),
(int) TimeUnit.MILLISECONDS.toSeconds(zookeeperSessionTimeoutMs), this.orderedExecutor);
localZkConnectionSvc.start(new ZookeeperSessionExpiredHandler() {
@Override
public void onSessionExpired() {
try {
localZkCache.getZooKeeper().close();
} catch (InterruptedException e) {
log.warn("Failed to shutdown ZooKeeper gracefully {}", e.getMessage(), e);
}
}
```
If we assume it was caused by zk session expires, does it means that connection with zk cluster has been broken?
What should we do in this situation, just restart the service or any other suggenstions?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] aloyszhang edited a comment on issue #7476: Discovery service can't find active broker
Posted by GitBox <gi...@apache.org>.
aloyszhang edited a comment on issue #7476:
URL: https://github.com/apache/pulsar/issues/7476#issuecomment-668934910
Since not found the exactlly reason cause this issue. We bring a compromise solution， try get active brokers from zk one more time if `availableBrokers` in `localZkCache` is empty. If zk does have brokers information, then return.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] aloyszhang edited a comment on issue #7476: Discovery service can't find active broker
Posted by GitBox <gi...@apache.org>.
aloyszhang edited a comment on issue #7476:
URL: https://github.com/apache/pulsar/issues/7476#issuecomment-656712253
@sijie, Both `DiscoveryServiceServlet` and `BrokerDiscoveryProvider ` will be started if run discovery service .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on issue #7476: Discovery service can't find active broker
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #7476:
URL: https://github.com/apache/pulsar/issues/7476#issuecomment-686851539
@aloyszhang Can you contribute the change back?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui closed issue #7476: Discovery service can't find active broker
Posted by GitBox <gi...@apache.org>.
codelipenghui closed issue #7476:
URL: https://github.com/apache/pulsar/issues/7476
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] aloyszhang commented on issue #7476: Discovery service can't find active broker
Posted by GitBox <gi...@apache.org>.
aloyszhang commented on issue #7476:
URL: https://github.com/apache/pulsar/issues/7476#issuecomment-686855329
@sijie
OK, I'll add a PR later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] aloyszhang commented on issue #7476: Discovery service can't find active broker
Posted by GitBox <gi...@apache.org>.
aloyszhang commented on issue #7476:
URL: https://github.com/apache/pulsar/issues/7476#issuecomment-668934910
Since not found the exactlly reason cause this issue. We bring a compromise solution， try get active brokers from zk one more time if `availableBrokers` in `localZkCache` is empty.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] aloyszhang edited a comment on issue #7476: Discovery service can't find active broker
Posted by GitBox <gi...@apache.org>.
aloyszhang edited a comment on issue #7476:
URL: https://github.com/apache/pulsar/issues/7476#issuecomment-656712253
@sijie,
At the first time, I thoung it was caused by session expires, by no SessionExpired related log can be found.
```java
localZkConnectionSvc.start(new ZookeeperSessionExpiredHandler() {
@Override
public void onSessionExpired() {
log.error("Shutting down ZK sessions: {}", -1);
}
@Override
public void setWatcher(ZooKeeperSessionWatcher watcher) {
}
});
this.localZkCache = new LocalZooKeeperCache(localZkConnectionSvc.getLocalZooKeeper(),
(int) TimeUnit.MILLISECONDS.toSeconds(zookeeperSessionTimeoutMs), this.orderedExecutor);
localZkConnectionSvc.start(new ZookeeperSessionExpiredHandler() {
@Override
public void onSessionExpired() {
try {
localZkCache.getZooKeeper().close();
} catch (InterruptedException e) {
log.warn("Failed to shutdown ZooKeeper gracefully {}", e.getMessage(), e);
}
}
```
This problem has happended several time in our pulsar cluster.
And , both `DiscoveryServiceServlet` and `BrokerDiscoveryProvider ` will be started if run discovery service .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] aloyszhang edited a comment on issue #7476: Discovery service can't find active broker
Posted by GitBox <gi...@apache.org>.
aloyszhang edited a comment on issue #7476:
URL: https://github.com/apache/pulsar/issues/7476#issuecomment-656712253
@sijie, Thanks for you reply.
This problem has happened several times in our pulsar cluster. At the first time, I thought it was caused by session expires, but no SessionExpired related log can be found.
```java
localZkConnectionSvc.start(new ZookeeperSessionExpiredHandler() {
@Override
public void onSessionExpired() {
log.error("Shutting down ZK sessions: {}", -1);
}
@Override
public void setWatcher(ZooKeeperSessionWatcher watcher) {
}
});
this.localZkCache = new LocalZooKeeperCache(localZkConnectionSvc.getLocalZooKeeper(),
(int) TimeUnit.MILLISECONDS.toSeconds(zookeeperSessionTimeoutMs), this.orderedExecutor);
localZkConnectionSvc.start(new ZookeeperSessionExpiredHandler() {
@Override
public void onSessionExpired() {
try {
localZkCache.getZooKeeper().close();
} catch (InterruptedException e) {
log.warn("Failed to shutdown ZooKeeper gracefully {}", e.getMessage(), e);
}
}
```
If we assume it was caused by zk session expires, does it means that connection with zk cluster has been broken?
What should we do in this situation, just restart the service or any other suggenstions?
Also , both `DiscoveryServiceServlet` and `BrokerDiscoveryProvider ` will be started if run discovery service .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] aloyszhang edited a comment on issue #7476: Discovery service can't find active broker
Posted by GitBox <gi...@apache.org>.
aloyszhang edited a comment on issue #7476:
URL: https://github.com/apache/pulsar/issues/7476#issuecomment-655260885
Addtionally, search zookkeeper cluster , no error found.
And we dump the discovery process, found that `BrokerDiscoveryProvider` does not has availableBrokers in its `ZookeeperCacheLoader` but `DiscoveryServiceServlet` has the information.
It's sly since `DiscoveryServiceServlet` and `BrokerDiscoveryProvider` basically has the same logical on processing zk cache.
@codelipenghui @merlimat Any suggestions about this？
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] aloyszhang edited a comment on issue #7476: Discovery service can't find active broker
Posted by GitBox <gi...@apache.org>.
aloyszhang edited a comment on issue #7476:
URL: https://github.com/apache/pulsar/issues/7476#issuecomment-655260885
Addtionally, ZooKeeper has no error.
And we dump the discovery process, found that `BrokerDiscoveryProvider` does not has availableBrokers in its `ZookeeperCacheLoader` but `DiscoveryServiceServlet` has the information.
It's sly since `DiscoveryServiceServlet` and `BrokerDiscoveryProvider` basically has the same logical on processing zk cache.
@codelipenghui @merlimat Any suggestions about this？
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie commented on issue #7476: Discovery service can't find active broker
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #7476:
URL: https://github.com/apache/pulsar/issues/7476#issuecomment-655854425
@aloyszhang
No available broker usually happens when the session expires. Out of curiosity, why do you need to start a separate discovery service?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] aloyszhang edited a comment on issue #7476: Discovery service can't find active broker
Posted by GitBox <gi...@apache.org>.
aloyszhang edited a comment on issue #7476:
URL: https://github.com/apache/pulsar/issues/7476#issuecomment-656712253
@sijie,
This problem has happened several times in our pulsar cluster. At the first time, I thought it was caused by session expires, but no SessionExpired related log can be found.
```java
localZkConnectionSvc.start(new ZookeeperSessionExpiredHandler() {
@Override
public void onSessionExpired() {
log.error("Shutting down ZK sessions: {}", -1);
}
@Override
public void setWatcher(ZooKeeperSessionWatcher watcher) {
}
});
this.localZkCache = new LocalZooKeeperCache(localZkConnectionSvc.getLocalZooKeeper(),
(int) TimeUnit.MILLISECONDS.toSeconds(zookeeperSessionTimeoutMs), this.orderedExecutor);
localZkConnectionSvc.start(new ZookeeperSessionExpiredHandler() {
@Override
public void onSessionExpired() {
try {
localZkCache.getZooKeeper().close();
} catch (InterruptedException e) {
log.warn("Failed to shutdown ZooKeeper gracefully {}", e.getMessage(), e);
}
}
```
If we assume it was caused by zk session expires, does it means that connection with zk cluster has been broken?
What should we do in this situation, just restart the service or any other suggenstions?
Also , both `DiscoveryServiceServlet` and `BrokerDiscoveryProvider ` will be started if run discovery service .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] aloyszhang edited a comment on issue #7476: Discovery service can't find active broker
Posted by GitBox <gi...@apache.org>.
aloyszhang edited a comment on issue #7476:
URL: https://github.com/apache/pulsar/issues/7476#issuecomment-668934910
Since not found the exactlly reason cause this issue. We bring a compromise solution， try get active brokers from zk one more time if `availableBrokers` in `localZkCache` is empty. If zk does have brokers information, then return. @sijie
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] aloyszhang commented on issue #7476: Discovery service can't find active broker
Posted by GitBox <gi...@apache.org>.
aloyszhang commented on issue #7476:
URL: https://github.com/apache/pulsar/issues/7476#issuecomment-656712253
@sijie, Both `DiscoveryServiceServlet` and `BrokerDiscoveryProvider ` will be started if runs discovery service .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org