You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2019/02/07 04:22:40 UTC
[GitHub] nkurihar commented on issue #3535: Broker stopped with a lot of
LedgerFencedException
nkurihar commented on issue #3535: Broker stopped with a lot of LedgerFencedException
URL: https://github.com/apache/pulsar/issues/3535#issuecomment-461283589
@sijie Thanks for your help!
> I think the exception should be self-recovered. did it self-recover?
No.
After ZK session expired, Broker becoame down.
```
01:47:09.961 [pulsar-ordered-OrderedExecutor-4-0-SendThread(zk101.pulsar.xxx.xx:2181)] WARN org.apache.zookeeper.ClientCnxn - Unable to reconnect to ZooKeeper service, session 0x3048c994034e22a has expired
01:47:09.961 [pulsar-ordered-OrderedExecutor-4-0-SendThread(zk101.pulsar.xxx.xx:2181)] INFO org.apache.zookeeper.ClientCnxn - Unable to reconnect to ZooKeeper service, session 0x3048c994034e22a has expired, closing socket connection
01:47:09.961 [pulsar-ordered-OrderedExecutor-4-0-EventThread] INFO o.a.p.zookeeper.ZooKeeperDataCache - [State:CLOSED sessionid:0x3048c994034e22a local:/xx.xx.xx.xx:37204 remoteserver:zk101.pulsar.xxx.xx/xx.xx.xx.xx:2181 lastZxid:171918958881 xid:7602221 sent:7602223 recv:8325363 queuedpkts:73 pendingresp:0 queuedevents:1] Received ZooKeeper watch event: WatchedEvent state:Expired type:None path:null
01:47:09.961 [pulsar-ordered-OrderedExecutor-4-0-EventThread] WARN o.a.p.zookeeper.LocalZooKeeperCache - Lost connection from local ZK. Invalidating the whole cache.
01:47:09.963 [pulsar-ordered-OrderedExecutor-4-0-EventThread] INFO o.a.p.z.ZooKeeperChildrenCache - [State:CLOSED sessionid:0x3048c994034e22a local:xx.xx.xx.xx/xx.xx.xx.xx:37204 remoteserver:zk101.pulsar.xxx.xx/xx.xx.xx.xx:2181 lastZxid:171918958881 xid:7602221 sent:7602223 recv:8325363 queuedpkts:73 pendingresp:0 queuedevents:1] Received ZooKeeper watch event: WatchedEvent state:Expired type:None path:null
01:47:09.963 [pulsar-ordered-OrderedExecutor-4-0-EventThread] WARN o.a.p.zookeeper.LocalZooKeeperCache - Lost connection from local ZK. Invalidating the whole cache.
01:47:09.963 [pulsar-ordered-OrderedExecutor-4-0-EventThread] INFO o.a.p.z.ZooKeeperChildrenCache - [State:CLOSED sessionid:0x3048c994034e22a local:xx.xx.xx.xx/xx.xx.xx.xx:37204 remoteserver:zk101.pulsar.xxx.xx/xx.xx.xx.xx:2181 lastZxid:171918958881 xid:7602221 sent:7602223 recv:8325363 queuedpkts:73 pendingresp:0 queuedevents:1] Received ZooKeeper watch event: WatchedEvent state:Expired type:None path:null
01:47:09.963 [pulsar-ordered-OrderedExecutor-4-0-EventThread] WARN o.a.p.zookeeper.LocalZooKeeperCache - Lost connection from local ZK. Invalidating the whole cache.
01:47:09.963 [pulsar-ordered-OrderedExecutor-4-0-EventThread] INFO o.a.p.zookeeper.ZooKeeperDataCache - [State:CLOSED sessionid:0x3048c994034e22a local:xx.xx.xx.xx/xx.xx.xx.xx:37204 remoteserver:zk101.pulsar.xxx.xx/xx.xx.xx.xx:2181 lastZxid:171918958881 xid:7602221 sent:7602223 recv:8325363 queuedpkts:73 pendingresp:0 queuedevents:1] Received ZooKeeper watch event: WatchedEvent state:Expired type:None path:null
01:47:09.963 [pulsar-ordered-OrderedExecutor-4-0-EventThread] WARN o.a.p.zookeeper.LocalZooKeeperCache - Lost connection from local ZK. Invalidating the whole cache.
01:47:09.963 [pulsar-ordered-OrderedExecutor-4-0-EventThread] INFO o.a.p.zookeeper.ZooKeeperDataCache - [State:CLOSED sessionid:0x3048c994034e22a local:xx.xx.xx.xx/xx.xx.xx.xx:37204 remoteserver:zk101.pulsar.xxx.xx/xx.xx.xx.xx:2181 lastZxid:171918958881 xid:7602221 sent:7602223 recv:8325363 queuedpkts:73 pendingresp:0 queuedevents:1] Received ZooKeeper watch event: WatchedEvent state:Expired type:None path:null
01:47:09.963 [pulsar-ordered-OrderedExecutor-4-0-EventThread] WARN o.a.p.zookeeper.LocalZooKeeperCache - Lost connection from local ZK. Invalidating the whole cache.
01:47:09.963 [pulsar-ordered-OrderedExecutor-4-0-EventThread] ERROR o.a.b.zookeeper.ZooKeeperWatcherBase - ZooKeeper client connection to the ZooKeeper server has expired!
01:47:09.963 [pulsar-ordered-OrderedExecutor-4-0-EventThread] INFO o.a.b.zookeeper.ZooKeeperClient - ZooKeeper session 3048c994034e22a is expired from zk100.pulsar.xxx.xx,zk101.pulsar.xxx.xx,zk102.pulsar.xxx.xx,zk103.pulsar.xxx.xx,zk104.pulsar.xxx.xx.
01:47:09.963 [pulsar-ordered-OrderedExecutor-4-0-EventThread] INFO o.a.p.z.ZooKeeperSessionWatcher - Received zookeeper notification, eventType=None, eventState=Expired
01:47:09.963 [pulsar-ordered-OrderedExecutor-4-0-EventThread] ERROR o.a.p.z.ZooKeeperSessionWatcher - ZooKeeper session already expired, invoking shutdown
01:47:09.963 [pulsar-zk-session-watcher-5-1] INFO o.a.p.z.ZooKeeperSessionWatcher - ZooKeeperSessionWatcher interrupted
01:47:09.963 [pulsar-ordered-OrderedExecutor-4-0-EventThread] INFO o.a.p.b.MessagingServiceShutdownHook - Invoking Runtime.halt(-1)
01:47:09.964 [ZKC-connect-executor-0] INFO o.a.b.zookeeper.ZooKeeperClient - Reconnecting zookeeper zk100.pulsar.xxx.xx,zk101.pulsar.xxx.xx,zk102.pulsar.xxx.xx,zk103.pulsar.xxx.xx,zk104.pulsar.xxx.xx.
01:47:09.966 [ZKC-connect-executor-0] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=zk100.pulsar.xxx.xx,zk101.pulsar.xxx.xx,zk102.pulsar.xxx.xx,zk103.pulsar.xxx.xx,zk104.pulsar.xxx.xx sessionTimeout=30000 watcher=org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase@1c7f3bee
(Broker became down and not self-recovered)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services