You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Jens Rantil <je...@tink.se> on 2016/08/25 19:51:17 UTC
Cleaning up giant znode
Hi,
A code mistake lead to us to write a lot of znodes with very random names
under the same directory/hierarchy in our Zookeeper ensemble.
Unfortunately, I am reaching the packet limit when trying to list the
nodes, so I can't delete them.
Does anyone have any idea how I could clean up these nodes?
Thanks,
Jens
--
Jens Rantil
Backend Developer @ Tink
Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
For urgent matters you can reach me at +46-708-84 18 32.
Re: Cleaning up giant znode
Posted by Jens Rantil <je...@tink.se>.
Hi,
Since we had so many I znodes I was afraid of doing a rolling restart of
the cluster. We generally have a lot of znodes and I've had issues with
rolling restarts (container znodes will fix this for us).
My woraround was this: I installed binutils' `strings` command on the
server and extracted the znode names from the snapshot file. I then
iterated over those names to delete the znodes in question. Worked like a
charm and avoided a restart.
Thanks for your input. Let's hope this never happens again :-)
Cheers,
Jens
On Thursday, August 25, 2016, Dan Benediktson
<db...@twitter.com.invalid> wrote:
> Yes, IIRC, the packet limit is enforced differently on server side and
> client side: the server side imposes it per-node that it sends, while the
> client imposes it for the whole response message. Since your problem is not
> a single big node, but a long list of nodes, I expect if you just override
> jute.maxbuffer on the client side to a big enough value (using the system
> property when launching your client), you should be able to list the nodes
> successfully.
>
> Dan
>
> On Thu, Aug 25, 2016 at 1:19 PM, Galo Navarro <anglorvaroa@gmail.com
> <javascript:;>> wrote:
>
> > Have you tried raising jute.maxbuffer [1]? That might work around the
> > listing issue and let you delete children individually.
> >
> > [1]: https://zookeeper.apache.org/doc/r3.4.8/zookeeperAdmin.html#
> > Experimental+Options%2FFeatures
> >
> > Cheers,
> > Galo
> >
> > On 25 August 2016 at 21:51, Jens Rantil <jens.rantil@tink.se
> <javascript:;>> wrote:
> >
> > > Hi,
> > >
> > > A code mistake lead to us to write a lot of znodes with very random
> names
> > > under the same directory/hierarchy in our Zookeeper ensemble.
> > > Unfortunately, I am reaching the packet limit when trying to list the
> > > nodes, so I can't delete them.
> > >
> > > Does anyone have any idea how I could clean up these nodes?
> > >
> > > Thanks,
> > > Jens
> > > --
> > >
> > > Jens Rantil
> > > Backend Developer @ Tink
> > >
> > > Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
> > > For urgent matters you can reach me at +46-708-84 18 32.
> > >
> >
>
--
Jens Rantil
Backend engineer
Tink AB
Email: jens.rantil@tink.se
Phone: +46 708 84 18 32
Web: www.tink.se
Facebook <https://www.facebook.com/#!/tink.se> Linkedin
<http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo&trkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary>
Twitter <https://twitter.com/tink>
Re: Cleaning up giant znode
Posted by Dan Benediktson <db...@twitter.com.INVALID>.
Yes, IIRC, the packet limit is enforced differently on server side and
client side: the server side imposes it per-node that it sends, while the
client imposes it for the whole response message. Since your problem is not
a single big node, but a long list of nodes, I expect if you just override
jute.maxbuffer on the client side to a big enough value (using the system
property when launching your client), you should be able to list the nodes
successfully.
Dan
On Thu, Aug 25, 2016 at 1:19 PM, Galo Navarro <an...@gmail.com> wrote:
> Have you tried raising jute.maxbuffer [1]? That might work around the
> listing issue and let you delete children individually.
>
> [1]: https://zookeeper.apache.org/doc/r3.4.8/zookeeperAdmin.html#
> Experimental+Options%2FFeatures
>
> Cheers,
> Galo
>
> On 25 August 2016 at 21:51, Jens Rantil <je...@tink.se> wrote:
>
> > Hi,
> >
> > A code mistake lead to us to write a lot of znodes with very random names
> > under the same directory/hierarchy in our Zookeeper ensemble.
> > Unfortunately, I am reaching the packet limit when trying to list the
> > nodes, so I can't delete them.
> >
> > Does anyone have any idea how I could clean up these nodes?
> >
> > Thanks,
> > Jens
> > --
> >
> > Jens Rantil
> > Backend Developer @ Tink
> >
> > Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
> > For urgent matters you can reach me at +46-708-84 18 32.
> >
>
Re: Cleaning up giant znode
Posted by Galo Navarro <an...@gmail.com>.
Have you tried raising jute.maxbuffer [1]? That might work around the
listing issue and let you delete children individually.
[1]: https://zookeeper.apache.org/doc/r3.4.8/zookeeperAdmin.html#
Experimental+Options%2FFeatures
Cheers,
Galo
On 25 August 2016 at 21:51, Jens Rantil <je...@tink.se> wrote:
> Hi,
>
> A code mistake lead to us to write a lot of znodes with very random names
> under the same directory/hierarchy in our Zookeeper ensemble.
> Unfortunately, I am reaching the packet limit when trying to list the
> nodes, so I can't delete them.
>
> Does anyone have any idea how I could clean up these nodes?
>
> Thanks,
> Jens
> --
>
> Jens Rantil
> Backend Developer @ Tink
>
> Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
> For urgent matters you can reach me at +46-708-84 18 32.
>