You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "Kommu, Vinodh K." <vk...@dtcc.com> on 2020/04/01 12:49:59 UTC

RE: Unable to delete zookeeper queue

Hi,

Does anyone know a working solution to delete zookeeper queue data? Please help!!


Regards,
Vinodh

From: Kommu, Vinodh K.
Sent: Tuesday, March 31, 2020 12:55 PM
To: solr-user@lucene.apache.org
Subject: Unable to delete zookeeper queue

All,

For some reason one of our zookeeper queue was filled with way bigger number so when I tried to delete queues with "rmr /overseer/queue" command, it's throwing - Packet len19029055 is out of range! exception. Later I have increased maxbuffer size to 50M and tried the same rmr command but still getting following error. Since the queues are not getting deleted, solr cluster status is not healthy which apparently marks all replicas as down even nodes are up & running. Looks like it is a known bug with zookeeper. Is there a way to delete zookeeper queues forcefully?

Error snippet:

Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /overseer/queue
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1500)
        at org.apache.zookeeper.ZKUtil.listSubTreeBFS(ZKUtil.java:114)
        at org.apache.zookeeper.ZKUtil.deleteRecursive(ZKUtil.java:49)
        at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:703)
        at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:588)
        at org.apache.zookeeper.ZooKeeperMain.executeLine(ZooKeeperMain.java:360)
        at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:323)
        at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:282)


Queue output:

get /overseer/queue
null
cZxid = 0x300000017
ctime = Wed Feb 06 22:29:14 EST 2019
mZxid = 0x300000017
mtime = Wed Feb 06 22:29:14 EST 2019
pZxid = 0x3400001869
cversion = 1420613
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 0
numChildren = 1119355


Regards,
Vinodh

DTCC DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify us immediately and delete the email and any attachments from your system. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email.

Re: Unable to delete zookeeper queue

Posted by Jörn Franke <jo...@gmail.com>.
Maybe you need I inc on zk server and zk client Jute Max bufffer to execute this . You can better ask the ZK mailing list 

> Am 01.04.2020 um 14:53 schrieb Kommu, Vinodh K. <vk...@dtcc.com>:
> 
> Hi,
> 
> Does anyone know a working solution to delete zookeeper queue data? Please help!!
> 
> 
> Regards,
> Vinodh
> 
> From: Kommu, Vinodh K.
> Sent: Tuesday, March 31, 2020 12:55 PM
> To: solr-user@lucene.apache.org
> Subject: Unable to delete zookeeper queue
> 
> All,
> 
> For some reason one of our zookeeper queue was filled with way bigger number so when I tried to delete queues with "rmr /overseer/queue" command, it's throwing - Packet len19029055 is out of range! exception. Later I have increased maxbuffer size to 50M and tried the same rmr command but still getting following error. Since the queues are not getting deleted, solr cluster status is not healthy which apparently marks all replicas as down even nodes are up & running. Looks like it is a known bug with zookeeper. Is there a way to delete zookeeper queues forcefully?
> 
> Error snippet:
> 
> Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /overseer/queue
>        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
>        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1500)
>        at org.apache.zookeeper.ZKUtil.listSubTreeBFS(ZKUtil.java:114)
>        at org.apache.zookeeper.ZKUtil.deleteRecursive(ZKUtil.java:49)
>        at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:703)
>        at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:588)
>        at org.apache.zookeeper.ZooKeeperMain.executeLine(ZooKeeperMain.java:360)
>        at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:323)
>        at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:282)
> 
> 
> Queue output:
> 
> get /overseer/queue
> null
> cZxid = 0x300000017
> ctime = Wed Feb 06 22:29:14 EST 2019
> mZxid = 0x300000017
> mtime = Wed Feb 06 22:29:14 EST 2019
> pZxid = 0x3400001869
> cversion = 1420613
> dataVersion = 0
> aclVersion = 0
> ephemeralOwner = 0x0
> dataLength = 0
> numChildren = 1119355
> 
> 
> Regards,
> Vinodh
> 
> DTCC DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify us immediately and delete the email and any attachments from your system. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email.