You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2019/02/07 04:22:40 UTC

[GitHub] nkurihar commented on issue #3535: Broker stopped with a lot of LedgerFencedException

nkurihar commented on issue #3535: Broker stopped with a lot of LedgerFencedException
URL: https://github.com/apache/pulsar/issues/3535#issuecomment-461283589
 
 
   @sijie Thanks for your help!
   
   > I think the exception should be self-recovered. did it self-recover?
   
   No.
   After ZK session expired, Broker becoame down.
   ```
   01:47:09.961 [pulsar-ordered-OrderedExecutor-4-0-SendThread(zk101.pulsar.xxx.xx:2181)] WARN  org.apache.zookeeper.ClientCnxn      - Unable to reconnect to ZooKeeper service, session 0x3048c994034e22a has expired
   01:47:09.961 [pulsar-ordered-OrderedExecutor-4-0-SendThread(zk101.pulsar.xxx.xx:2181)] INFO  org.apache.zookeeper.ClientCnxn      - Unable to reconnect to ZooKeeper service, session 0x3048c994034e22a has expired, closing socket connection
   01:47:09.961 [pulsar-ordered-OrderedExecutor-4-0-EventThread] INFO  o.a.p.zookeeper.ZooKeeperDataCache   - [State:CLOSED sessionid:0x3048c994034e22a local:/xx.xx.xx.xx:37204 remoteserver:zk101.pulsar.xxx.xx/xx.xx.xx.xx:2181 lastZxid:171918958881 xid:7602221 sent:7602223 recv:8325363 queuedpkts:73 pendingresp:0 queuedevents:1] Received ZooKeeper watch event: WatchedEvent state:Expired type:None path:null
   01:47:09.961 [pulsar-ordered-OrderedExecutor-4-0-EventThread] WARN  o.a.p.zookeeper.LocalZooKeeperCache  - Lost connection from local ZK. Invalidating the whole cache.
   01:47:09.963 [pulsar-ordered-OrderedExecutor-4-0-EventThread] INFO  o.a.p.z.ZooKeeperChildrenCache       - [State:CLOSED sessionid:0x3048c994034e22a local:xx.xx.xx.xx/xx.xx.xx.xx:37204 remoteserver:zk101.pulsar.xxx.xx/xx.xx.xx.xx:2181 lastZxid:171918958881 xid:7602221 sent:7602223 recv:8325363 queuedpkts:73 pendingresp:0 queuedevents:1] Received ZooKeeper watch event: WatchedEvent state:Expired type:None path:null
   01:47:09.963 [pulsar-ordered-OrderedExecutor-4-0-EventThread] WARN  o.a.p.zookeeper.LocalZooKeeperCache  - Lost connection from local ZK. Invalidating the whole cache.
   01:47:09.963 [pulsar-ordered-OrderedExecutor-4-0-EventThread] INFO  o.a.p.z.ZooKeeperChildrenCache       - [State:CLOSED sessionid:0x3048c994034e22a local:xx.xx.xx.xx/xx.xx.xx.xx:37204 remoteserver:zk101.pulsar.xxx.xx/xx.xx.xx.xx:2181 lastZxid:171918958881 xid:7602221 sent:7602223 recv:8325363 queuedpkts:73 pendingresp:0 queuedevents:1] Received ZooKeeper watch event: WatchedEvent state:Expired type:None path:null
   01:47:09.963 [pulsar-ordered-OrderedExecutor-4-0-EventThread] WARN  o.a.p.zookeeper.LocalZooKeeperCache  - Lost connection from local ZK. Invalidating the whole cache.
   01:47:09.963 [pulsar-ordered-OrderedExecutor-4-0-EventThread] INFO  o.a.p.zookeeper.ZooKeeperDataCache   - [State:CLOSED sessionid:0x3048c994034e22a local:xx.xx.xx.xx/xx.xx.xx.xx:37204 remoteserver:zk101.pulsar.xxx.xx/xx.xx.xx.xx:2181 lastZxid:171918958881 xid:7602221 sent:7602223 recv:8325363 queuedpkts:73 pendingresp:0 queuedevents:1] Received ZooKeeper watch event: WatchedEvent state:Expired type:None path:null
   01:47:09.963 [pulsar-ordered-OrderedExecutor-4-0-EventThread] WARN  o.a.p.zookeeper.LocalZooKeeperCache  - Lost connection from local ZK. Invalidating the whole cache.
   01:47:09.963 [pulsar-ordered-OrderedExecutor-4-0-EventThread] INFO  o.a.p.zookeeper.ZooKeeperDataCache   - [State:CLOSED sessionid:0x3048c994034e22a local:xx.xx.xx.xx/xx.xx.xx.xx:37204 remoteserver:zk101.pulsar.xxx.xx/xx.xx.xx.xx:2181 lastZxid:171918958881 xid:7602221 sent:7602223 recv:8325363 queuedpkts:73 pendingresp:0 queuedevents:1] Received ZooKeeper watch event: WatchedEvent state:Expired type:None path:null
   01:47:09.963 [pulsar-ordered-OrderedExecutor-4-0-EventThread] WARN  o.a.p.zookeeper.LocalZooKeeperCache  - Lost connection from local ZK. Invalidating the whole cache.
   01:47:09.963 [pulsar-ordered-OrderedExecutor-4-0-EventThread] ERROR o.a.b.zookeeper.ZooKeeperWatcherBase - ZooKeeper client connection to the ZooKeeper server has expired!
   01:47:09.963 [pulsar-ordered-OrderedExecutor-4-0-EventThread] INFO  o.a.b.zookeeper.ZooKeeperClient      - ZooKeeper session 3048c994034e22a is expired from zk100.pulsar.xxx.xx,zk101.pulsar.xxx.xx,zk102.pulsar.xxx.xx,zk103.pulsar.xxx.xx,zk104.pulsar.xxx.xx.
   01:47:09.963 [pulsar-ordered-OrderedExecutor-4-0-EventThread] INFO  o.a.p.z.ZooKeeperSessionWatcher      - Received zookeeper notification, eventType=None, eventState=Expired
   01:47:09.963 [pulsar-ordered-OrderedExecutor-4-0-EventThread] ERROR o.a.p.z.ZooKeeperSessionWatcher      - ZooKeeper session already expired, invoking shutdown
   01:47:09.963 [pulsar-zk-session-watcher-5-1] INFO  o.a.p.z.ZooKeeperSessionWatcher      - ZooKeeperSessionWatcher interrupted
   01:47:09.963 [pulsar-ordered-OrderedExecutor-4-0-EventThread] INFO  o.a.p.b.MessagingServiceShutdownHook - Invoking Runtime.halt(-1)
   01:47:09.964 [ZKC-connect-executor-0] INFO  o.a.b.zookeeper.ZooKeeperClient      - Reconnecting zookeeper zk100.pulsar.xxx.xx,zk101.pulsar.xxx.xx,zk102.pulsar.xxx.xx,zk103.pulsar.xxx.xx,zk104.pulsar.xxx.xx.
   01:47:09.966 [ZKC-connect-executor-0] INFO  org.apache.zookeeper.ZooKeeper       - Initiating client connection, connectString=zk100.pulsar.xxx.xx,zk101.pulsar.xxx.xx,zk102.pulsar.xxx.xx,zk103.pulsar.xxx.xx,zk104.pulsar.xxx.xx sessionTimeout=30000 watcher=org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase@1c7f3bee
   
   (Broker became down and not self-recovered)
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services