You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@curator.apache.org by Jens Rantil <je...@tink.se> on 2015/09/13 18:41:04 UTC

InterProcessSemaphoreMutex not cleaning up old znodes

Hi again,

I have a distributed application which has InterProcessSemaphoreMutexes which
occasionally are taken per user. To do this, the mutex path is appended
with a user's unique ID (example: /mylock/{userid}). All is fine and dandy
and this generally works. However, I've noticed that the paths given to the
InterProcessSemaphoreMutexes aren't cleaned up after I'm done with the
locks.

This means, my Zookeeper ensemble has every single one of my previous
mutexes held in memory. When I list my znodes I see:
/mylock/1
/mylock/2
...
/mylock/XXX

Given that I generally only take ~20 locks at a time it feels unnecessary
to keep all these in memory. We have had to increase initLimit in Zookeeper
configuration, most certainly due to all of these znodes.

Two questions:

- Is there a reason for keeping these znodes hanging around?
- Given that I need to use mutexes, what are the best practise for
taking locks per user like this? Hash the user IDs and bucket them into a
limitted number of mutexes (enough to avoid contention)? Have a separate
cleaning thread that cleans up old znodes? How are you guys handling this?

Thanks,
Jens

--
Jens Rantil
Backend engineer
Tink AB

Email: jens.rantil@tink.se
Phone: +46 708 84 18 32
Web: www.tink.se

Facebook <https://www.facebook.com/#!/tink.se> Linkedin
<http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo&trkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary>
Twitter <https://twitter.com/tink>

Re: InterProcessSemaphoreMutex not cleaning up old znodes

Posted by Jens Rantil <je...@tink.se>.

Thanks, Jordan. The ChildReaper is exactly what I was looking for.

I'll hold off regarding upgrading of Zookeeper and Curator for now.

Cheers,
Jens

On Sun, Sep 13, 2015 at 10:27 PM, Jordan Zimmerman <
jordan@jordanzimmerman.com> wrote:

> This has long been an issue in ZooKeeper. The good news is that there are
> solutions:
>
> * Current releases: Curator has utilities for cleaning up old parent
> nodes. Look at Reaper and ChildReaper
> * New releases: ZooKeeper 3.5.1 + Curator 2.9.0 (just released) support
> the new Container node feature. Curator 2.9.0 will automatically create
> container nodes which are automatically cleaned up when empty.
>
> -Jordan
>
>
> On September 13, 2015 at 11:41:41 AM, Jens Rantil (jens.rantil@tink.se)
> wrote:
>
> Hi again,
>
> I have a distributed application which has InterProcessSemaphoreMutexes which
> occasionally are taken per user. To do this, the mutex path is appended
> with a user's unique ID (example: /mylock/{userid}). All is fine and
> dandy and this generally works. However, I've noticed that the paths given
> to the InterProcessSemaphoreMutexes aren't cleaned up after I'm done with
> the locks.
>
> This means, my Zookeeper ensemble has every single one of my previous
> mutexes held in memory. When I list my znodes I see:
> /mylock/1
> /mylock/2
> ...
> /mylock/XXX
>
> Given that I generally only take ~20 locks at a time it feels unnecessary
> to keep all these in memory. We have had to increase initLimit in
> Zookeeper configuration, most certainly due to all of these znodes.
>
> Two questions:
>
>    - Is there a reason for keeping these znodes hanging around?
>    - Given that I need to use mutexes, what are the best practise for
>    taking locks per user like this? Hash the user IDs and bucket them into a
>    limitted number of mutexes (enough to avoid contention)? Have a separate
>    cleaning thread that cleans up old znodes? How are you guys handling this?
>
> Thanks,
> Jens
>
> --
> Jens Rantil
> Backend engineer
> Tink AB
>
> Email: jens.rantil@tink.se
> Phone: +46 708 84 18 32
> Web: www.tink.se
>
> Facebook <https://www.facebook.com/#!/tink.se> Linkedin
> <http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo&trkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary>
>  Twitter <https://twitter.com/tink>
>
>


-- 
Jens Rantil
Backend engineer
Tink AB

Email: jens.rantil@tink.se
Phone: +46 708 84 18 32
Web: www.tink.se

Facebook <https://www.facebook.com/#!/tink.se> Linkedin
<http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo&trkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary>
 Twitter <https://twitter.com/tink>

Re: InterProcessSemaphoreMutex not cleaning up old znodes

Posted by Jens Rantil <je...@tink.se>.

Jordan,

I see. That explains things.

Thanks,
Jens

On Thu, Sep 17, 2015 at 4:45 PM, Jordan Zimmerman <
jordan@jordanzimmerman.com> wrote:

> Jens,
>
> Reaper and ChildReaper are now deprecated. Moving forward, Curator will
> automatically use container nodes which are now supported directly by
> ZooKeeper (I wrote the feature). Reaper/ChildReaper was always a hack to
> work around the situation.
>
> -JZ
>
>
>
> On September 17, 2015 at 9:43:03 AM, Jens Rantil (jens.rantil@tink.se)
> wrote:
>
> Hi again Jordan,
>
> Your input is highly appreciated. Thanks!
>
> I assume you mean I would do something like
>
>    1. myReaper = new ChildReaper(client, "/mylock")
>    2. myReaper.start()
>    3. mutex = new InterProcessSemaphoreMutex(client, "/mylock/user123");
>    4. myReaper.addPath("/mylock/user123")
>    5. // Use the mutex...
>
> However, when reading the implementation of ChildReaper (version 2.7.1)
> it looks like the paths member field never removes "/mylock/user123" if
> it has been reaped by the reaper for "/mylock". This would essentially
> become a memory leak if I have a lot of users, right? Or have I
> misunderstood something?
>
> Generally this isn't a pretty abstraction since a ChildReaper user (me)
> must essentially know how the internals of a recipe to know if it creates
> additional hierarchies or not. A different approach would be to add a
> `Reapable` interface to all recipe classes and have them implement how they
> would reap themselves. Possibly they could simply delegate to a
> RecursiveChildReaper that supports a configurable cleanup level (1 for
> basic locks, 2 for InterProcessSemaphoreMutexes that has two level znode
> hierarchies).
>
> Thanks,
> Jens
>
> On Thu, Sep 17, 2015 at 3:38 PM, Jordan Zimmerman <
> jordan@jordanzimmerman.com> wrote:
> >
> > You’d need to add each level to the ChildReaper. You can also move to
> ZooKeeper 3.5.1-alpha and Curator 2.9.0 and you’ll get automatic cleanup
> (via Container nodes).
> >
> > -Jordan
> >
> >
> >
> > On September 17, 2015 at 8:11:49 AM, Jens Rantil (jens.rantil@tink.se)
> wrote:
> >
> > Hi again,
> >
> > A follow-up regarding this: I also have a couple of
> InterProcessSemaphoreMutexes that I'd like to clean up and since they are
> using using a nested hierachy (containing "locks" and "leases"), they
> aren't being cleaned up my ChildReaper. Is there a way to clean these up
> automatically?
> >
> > Thanks,
> > Jens
> >
> > On Mon, Sep 14, 2015 at 3:47 PM, Jordan Zimmerman <
> jordan@jordanzimmerman.com> wrote:
> >>
> >> ChildReaper uses a Reaper internally. So, just use whichever one works
> for you.
> >>
> >> -Jordan
> >>
> >>
> >>
> >> On September 14, 2015 at 8:30:19 AM, Jens Rantil (jens.rantil@tink.se)
> wrote:
> >>
> >> Hi again,
> >>
> >> A small follow-up question. What are the pros and cons of using Reaper
> vs. ChildReaper? It seems to me that using ChildReaper will involve a lot
> less intrusive code. Are there performance implications of one or the other?
> >>
> >> Thanks,
> >> Jens
> >>
> >> On Sun, Sep 13, 2015 at 10:27 PM, Jordan Zimmerman <
> jordan@jordanzimmerman.com> wrote:
> >>>
> >>> This has long been an issue in ZooKeeper. The good news is that there
> are solutions:
> >>>
> >>> * Current releases: Curator has utilities for cleaning up old parent
> nodes. Look at Reaper and ChildReaper
> >>> * New releases: ZooKeeper 3.5.1 + Curator 2.9.0 (just released)
> support the new Container node feature. Curator 2.9.0 will automatically
> create container nodes which are automatically cleaned up when empty.
> >>>
> >>> -Jordan
> >>>
> >>>
> >>> On September 13, 2015 at 11:41:41 AM, Jens Rantil (jens.rantil@tink.se)
> wrote:
> >>>
> >>> Hi again,
> >>>
> >>> I have a distributed application which has
> InterProcessSemaphoreMutexes which occasionally are taken per user. To do
> this, the mutex path is appended with a user's unique ID (example:
> /mylock/{userid}). All is fine and dandy and this generally works. However,
> I've noticed that the paths given to the InterProcessSemaphoreMutexes
> aren't cleaned up after I'm done with the locks.
> >>>
> >>> This means, my Zookeeper ensemble has every single one of my previous
> mutexes held in memory. When I list my znodes I see:
> >>> /mylock/1
> >>> /mylock/2
> >>> ...
> >>> /mylock/XXX
> >>>
> >>> Given that I generally only take ~20 locks at a time it feels
> unnecessary to keep all these in memory. We have had to increase initLimit
> in Zookeeper configuration, most certainly due to all of these znodes.
> >>>
> >>> Two questions:
> >>>
> >>> Is there a reason for keeping these znodes hanging around?
> >>> Given that I need to use mutexes, what are the best practise for
> taking locks per user like this? Hash the user IDs and bucket them into a
> limitted number of mutexes (enough to avoid contention)? Have a separate
> cleaning thread that cleans up old znodes? How are you guys handling this?
> >>>
> >>> Thanks,
> >>> Jens
> >>>
> >>> --
> >>> Jens Rantil
> >>> Backend engineer
> >>> Tink AB
> >>>
> >>> Email: jens.rantil@tink.se
> >>> Phone: +46 708 84 18 32
> >>> Web: www.tink.se
> >>>
> >>> Facebook Linkedin Twitter
> >>
> >>
> >>
> >>
> >> --
> >> Jens Rantil
> >> Backend engineer
> >> Tink AB
> >>
> >> Email: jens.rantil@tink.se
> >> Phone: +46 708 84 18 32
> >> Web: www.tink.se
> >>
> >> Facebook Linkedin Twitter
> >
> >
> >
> >
> > --
> > Jens Rantil
> > Backend engineer
> > Tink AB
> >
> > Email: jens.rantil@tink.se
> > Phone: +46 708 84 18 32
> > Web: www.tink.se
> >
> > Facebook Linkedin Twitter
>
>
>
>
> --
> Jens Rantil
> Backend engineer
> Tink AB
>
> Email: jens.rantil@tink.se
> Phone: +46 708 84 18 32
> Web: www.tink.se
>
> Facebook Linkedin Twitter
>
>


-- 
Jens Rantil
Backend engineer
Tink AB

Email: jens.rantil@tink.se
Phone: +46 708 84 18 32
Web: www.tink.se

Facebook <https://www.facebook.com/#!/tink.se> Linkedin
<http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo&trkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary>
 Twitter <https://twitter.com/tink>

Re: InterProcessSemaphoreMutex not cleaning up old znodes

Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.

Jens,

Reaper and ChildReaper are now deprecated. Moving forward, Curator will automatically use container nodes which are now supported directly by ZooKeeper (I wrote the feature). Reaper/ChildReaper was always a hack to work around the situation.

-JZ



On September 17, 2015 at 9:43:03 AM, Jens Rantil (jens.rantil@tink.se) wrote:

Hi again Jordan,

Your input is highly appreciated. Thanks!

I assume you mean I would do something like
myReaper = new ChildReaper(client, "/mylock")
myReaper.start()
mutex = new InterProcessSemaphoreMutex(client, "/mylock/user123");
myReaper.addPath("/mylock/user123")
// Use the mutex...
However, when reading the implementation of ChildReaper (version 2.7.1) it looks like the paths member field never removes "/mylock/user123" if it has been reaped by the reaper for "/mylock". This would essentially become a memory leak if I have a lot of users, right? Or have I misunderstood something?

Generally this isn't a pretty abstraction since a ChildReaper user (me) must essentially know how the internals of a recipe to know if it creates additional hierarchies or not. A different approach would be to add a `Reapable` interface to all recipe classes and have them implement how they would reap themselves. Possibly they could simply delegate to a RecursiveChildReaper that supports a configurable cleanup level (1 for basic locks, 2 for InterProcessSemaphoreMutexes that has two level znode hierarchies).

Thanks,
Jens

On Thu, Sep 17, 2015 at 3:38 PM, Jordan Zimmerman <jo...@jordanzimmerman.com> wrote:
>
> You’d need to add each level to the ChildReaper. You can also move to ZooKeeper 3.5.1-alpha and Curator 2.9.0 and you’ll get automatic cleanup (via Container nodes).
>
> -Jordan
>
>
>
> On September 17, 2015 at 8:11:49 AM, Jens Rantil (jens.rantil@tink.se) wrote:
>
> Hi again,
>
> A follow-up regarding this: I also have a couple of InterProcessSemaphoreMutexes that I'd like to clean up and since they are using using a nested hierachy (containing "locks" and "leases"), they aren't being cleaned up my ChildReaper. Is there a way to clean these up automatically?
>
> Thanks,
> Jens
>
> On Mon, Sep 14, 2015 at 3:47 PM, Jordan Zimmerman <jo...@jordanzimmerman.com> wrote:
>>
>> ChildReaper uses a Reaper internally. So, just use whichever one works for you.
>>
>> -Jordan
>>
>>
>>
>> On September 14, 2015 at 8:30:19 AM, Jens Rantil (jens.rantil@tink.se) wrote:
>>
>> Hi again,
>>
>> A small follow-up question. What are the pros and cons of using Reaper vs. ChildReaper? It seems to me that using ChildReaper will involve a lot less intrusive code. Are there performance implications of one or the other?
>>
>> Thanks,
>> Jens
>>
>> On Sun, Sep 13, 2015 at 10:27 PM, Jordan Zimmerman <jo...@jordanzimmerman.com> wrote:
>>>
>>> This has long been an issue in ZooKeeper. The good news is that there are solutions:
>>>
>>> * Current releases: Curator has utilities for cleaning up old parent nodes. Look at Reaper and ChildReaper
>>> * New releases: ZooKeeper 3.5.1 + Curator 2.9.0 (just released) support the new Container node feature. Curator 2.9.0 will automatically create container nodes which are automatically cleaned up when empty.
>>>
>>> -Jordan
>>>
>>>
>>> On September 13, 2015 at 11:41:41 AM, Jens Rantil (jens.rantil@tink.se) wrote:
>>>
>>> Hi again,
>>>
>>> I have a distributed application which has InterProcessSemaphoreMutexes which occasionally are taken per user. To do this, the mutex path is appended with a user's unique ID (example: /mylock/{userid}). All is fine and dandy and this generally works. However, I've noticed that the paths given to the InterProcessSemaphoreMutexes aren't cleaned up after I'm done with the locks.
>>>
>>> This means, my Zookeeper ensemble has every single one of my previous mutexes held in memory. When I list my znodes I see:
>>> /mylock/1
>>> /mylock/2
>>> ...
>>> /mylock/XXX
>>>
>>> Given that I generally only take ~20 locks at a time it feels unnecessary to keep all these in memory. We have had to increase initLimit in Zookeeper configuration, most certainly due to all of these znodes.
>>>
>>> Two questions:
>>>
>>> Is there a reason for keeping these znodes hanging around?
>>> Given that I need to use mutexes, what are the best practise for taking locks per user like this? Hash the user IDs and bucket them into a limitted number of mutexes (enough to avoid contention)? Have a separate cleaning thread that cleans up old znodes? How are you guys handling this?
>>>
>>> Thanks,
>>> Jens
>>>
>>> --
>>> Jens Rantil
>>> Backend engineer
>>> Tink AB
>>>
>>> Email: jens.rantil@tink.se
>>> Phone: +46 708 84 18 32
>>> Web: www.tink.se
>>>
>>> Facebook Linkedin Twitter
>>
>>
>>
>>
>> --
>> Jens Rantil
>> Backend engineer
>> Tink AB
>>
>> Email: jens.rantil@tink.se
>> Phone: +46 708 84 18 32
>> Web: www.tink.se
>>
>> Facebook Linkedin Twitter
>
>
>
>
> --
> Jens Rantil
> Backend engineer
> Tink AB
>
> Email: jens.rantil@tink.se
> Phone: +46 708 84 18 32
> Web: www.tink.se
>
> Facebook Linkedin Twitter




--
Jens Rantil
Backend engineer
Tink AB

Email: jens.rantil@tink.se
Phone: +46 708 84 18 32
Web: www.tink.se

Facebook Linkedin Twitter

Re: InterProcessSemaphoreMutex not cleaning up old znodes

Posted by Jens Rantil <je...@tink.se>.

Hi again Jordan,

Your input is highly appreciated. Thanks!

I assume you mean I would do something like

   1. myReaper = new ChildReaper(client, "/mylock")
   2. myReaper.start()
   3. mutex = new InterProcessSemaphoreMutex(client, "/mylock/user123");
   4. myReaper.addPath("/mylock/user123")
   5. // Use the mutex...

However, when reading the implementation of ChildReaper (version 2.7.1) it
looks like the paths member field never removes "/mylock/user123" if it has
been reaped by the reaper for "/mylock". This would essentially become a
memory leak if I have a lot of users, right? Or have I misunderstood
something?

Generally this isn't a pretty abstraction since a ChildReaper user (me)
must essentially know how the internals of a recipe to know if it creates
additional hierarchies or not. A different approach would be to add a
`Reapable` interface to all recipe classes and have them implement how they
would reap themselves. Possibly they could simply delegate to a
RecursiveChildReaper that supports a configurable cleanup level (1 for
basic locks, 2 for InterProcessSemaphoreMutexes that has two level znode
hierarchies).

Thanks,
Jens

On Thu, Sep 17, 2015 at 3:38 PM, Jordan Zimmerman <
jordan@jordanzimmerman.com> wrote:
>
> You’d need to add each level to the ChildReaper. You can also move to
ZooKeeper 3.5.1-alpha and Curator 2.9.0 and you’ll get automatic cleanup
(via Container nodes).
>
> -Jordan
>
>
>
> On September 17, 2015 at 8:11:49 AM, Jens Rantil (jens.rantil@tink.se)
wrote:
>
> Hi again,
>
> A follow-up regarding this: I also have a couple of
InterProcessSemaphoreMutexes that I'd like to clean up and since they are
using using a nested hierachy (containing "locks" and "leases"), they
aren't being cleaned up my ChildReaper. Is there a way to clean these up
automatically?
>
> Thanks,
> Jens
>
> On Mon, Sep 14, 2015 at 3:47 PM, Jordan Zimmerman <
jordan@jordanzimmerman.com> wrote:
>>
>> ChildReaper uses a Reaper internally. So, just use whichever one works
for you.
>>
>> -Jordan
>>
>>
>>
>> On September 14, 2015 at 8:30:19 AM, Jens Rantil (jens.rantil@tink.se)
wrote:
>>
>> Hi again,
>>
>> A small follow-up question. What are the pros and cons of using Reaper
vs. ChildReaper? It seems to me that using ChildReaper will involve a lot
less intrusive code. Are there performance implications of one or the other?
>>
>> Thanks,
>> Jens
>>
>> On Sun, Sep 13, 2015 at 10:27 PM, Jordan Zimmerman <
jordan@jordanzimmerman.com> wrote:
>>>
>>> This has long been an issue in ZooKeeper. The good news is that there
are solutions:
>>>
>>> * Current releases: Curator has utilities for cleaning up old parent
nodes. Look at Reaper and ChildReaper
>>> * New releases: ZooKeeper 3.5.1 + Curator 2.9.0 (just released) support
the new Container node feature. Curator 2.9.0 will automatically create
container nodes which are automatically cleaned up when empty.
>>>
>>> -Jordan
>>>
>>>
>>> On September 13, 2015 at 11:41:41 AM, Jens Rantil (jens.rantil@tink.se)
wrote:
>>>
>>> Hi again,
>>>
>>> I have a distributed application which has InterProcessSemaphoreMutexes
which occasionally are taken per user. To do this, the mutex path is
appended with a user's unique ID (example: /mylock/{userid}). All is fine
and dandy and this generally works. However, I've noticed that the paths
given to the InterProcessSemaphoreMutexes aren't cleaned up after I'm done
with the locks.
>>>
>>> This means, my Zookeeper ensemble has every single one of my previous
mutexes held in memory. When I list my znodes I see:
>>> /mylock/1
>>> /mylock/2
>>> ...
>>> /mylock/XXX
>>>
>>> Given that I generally only take ~20 locks at a time it feels
unnecessary to keep all these in memory. We have had to increase initLimit
in Zookeeper configuration, most certainly due to all of these znodes.
>>>
>>> Two questions:
>>>
>>> Is there a reason for keeping these znodes hanging around?
>>> Given that I need to use mutexes, what are the best practise for taking
locks per user like this? Hash the user IDs and bucket them into a limitted
number of mutexes (enough to avoid contention)? Have a separate cleaning
thread that cleans up old znodes? How are you guys handling this?
>>>
>>> Thanks,
>>> Jens
>>>
>>> --
>>> Jens Rantil
>>> Backend engineer
>>> Tink AB
>>>
>>> Email: jens.rantil@tink.se
>>> Phone: +46 708 84 18 32
>>> Web: www.tink.se
>>>
>>> Facebook Linkedin Twitter
>>
>>
>>
>>
>> --
>> Jens Rantil
>> Backend engineer
>> Tink AB
>>
>> Email: jens.rantil@tink.se
>> Phone: +46 708 84 18 32
>> Web: www.tink.se
>>
>> Facebook Linkedin Twitter
>
>
>
>
> --
> Jens Rantil
> Backend engineer
> Tink AB
>
> Email: jens.rantil@tink.se
> Phone: +46 708 84 18 32
> Web: www.tink.se
>
> Facebook Linkedin Twitter




--
Jens Rantil
Backend engineer
Tink AB

Email: jens.rantil@tink.se
Phone: +46 708 84 18 32
Web: www.tink.se

Facebook Linkedin Twitter

Re: InterProcessSemaphoreMutex not cleaning up old znodes

Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.

You’d need to add each level to the ChildReaper. You can also move to ZooKeeper 3.5.1-alpha and Curator 2.9.0 and you’ll get automatic cleanup (via Container nodes).

-Jordan

On September 17, 2015 at 8:11:49 AM, Jens Rantil (jens.rantil@tink.se) wrote:

Hi again,

A follow-up regarding this: I also have a couple of InterProcessSemaphoreMutexes that I'd like to clean up and since they are using using a nested hierachy (containing "locks" and "leases"), they aren't being cleaned up my ChildReaper. Is there a way to clean these up automatically?

Thanks,
Jens

On Mon, Sep 14, 2015 at 3:47 PM, Jordan Zimmerman <jo...@jordanzimmerman.com> wrote:
ChildReaper uses a Reaper internally. So, just use whichever one works for you.

-Jordan

On September 14, 2015 at 8:30:19 AM, Jens Rantil (jens.rantil@tink.se) wrote:

Hi again,

A small follow-up question. What are the pros and cons of using Reaper vs. ChildReaper? It seems to me that using ChildReaper will involve a lot less intrusive code. Are there performance implications of one or the other?

Thanks,
Jens

On Sun, Sep 13, 2015 at 10:27 PM, Jordan Zimmerman <jo...@jordanzimmerman.com> wrote:
This has long been an issue in ZooKeeper. The good news is that there are solutions:

* Current releases: Curator has utilities for cleaning up old parent nodes. Look at Reaper and ChildReaper
* New releases: ZooKeeper 3.5.1 + Curator 2.9.0 (just released) support the new Container node feature. Curator 2.9.0 will automatically create container nodes which are automatically cleaned up when empty.

-Jordan

On September 13, 2015 at 11:41:41 AM, Jens Rantil (jens.rantil@tink.se) wrote:

Hi again,

I have a distributed application which has InterProcessSemaphoreMutexes which occasionally are taken per user. To do this, the mutex path is appended with a user's unique ID (example: /mylock/{userid}). All is fine and dandy and this generally works. However, I've noticed that the paths given to the InterProcessSemaphoreMutexes aren't cleaned up after I'm done with the locks.

This means, my Zookeeper ensemble has every single one of my previous mutexes held in memory. When I list my znodes I see:
/mylock/1
/mylock/2
...
/mylock/XXX

Given that I generally only take ~20 locks at a time it feels unnecessary to keep all these in memory. We have had to increase initLimit in Zookeeper configuration, most certainly due to all of these znodes.

Two questions:
Is there a reason for keeping these znodes hanging around?
Given that I need to use mutexes, what are the best practise for taking locks per user like this? Hash the user IDs and bucket them into a limitted number of mutexes (enough to avoid contention)? Have a separate cleaning thread that cleans up old znodes? How are you guys handling this?
Thanks,
Jens

--
Jens Rantil
Backend engineer
Tink AB

Email: jens.rantil@tink.se
Phone: +46 708 84 18 32
Web: www.tink.se

Facebook Linkedin Twitter

--
Jens Rantil
Backend engineer
Tink AB

Email: jens.rantil@tink.se
Phone: +46 708 84 18 32
Web: www.tink.se

Facebook Linkedin Twitter

--
Jens Rantil
Backend engineer
Tink AB

Email: jens.rantil@tink.se
Phone: +46 708 84 18 32
Web: www.tink.se

Facebook Linkedin Twitter

Re: InterProcessSemaphoreMutex not cleaning up old znodes

Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.

ChildReaper uses a Reaper internally. So, just use whichever one works for you.

-Jordan

On September 14, 2015 at 8:30:19 AM, Jens Rantil (jens.rantil@tink.se) wrote:

Hi again,

Thanks,
Jens

On Sun, Sep 13, 2015 at 10:27 PM, Jordan Zimmerman <jo...@jordanzimmerman.com> wrote:
This has long been an issue in ZooKeeper. The good news is that there are solutions:

-Jordan

On September 13, 2015 at 11:41:41 AM, Jens Rantil (jens.rantil@tink.se) wrote:

Hi again,

I have a distributed application which has InterProcessSemaphoreMutexes which occasionally are taken per user. To do this, the mutex path is appended with a user's unique ID (example: /mylock/{userid}). All is fine and dandy and this generally works. However, I've noticed that the paths given to the InterProcessSemaphoreMutexes aren't cleaned up after I'm done with the locks.

This means, my Zookeeper ensemble has every single one of my previous mutexes held in memory. When I list my znodes I see:
/mylock/1
/mylock/2
...
/mylock/XXX

Given that I generally only take ~20 locks at a time it feels unnecessary to keep all these in memory. We have had to increase initLimit in Zookeeper configuration, most certainly due to all of these znodes.

--
Jens Rantil
Backend engineer
Tink AB

Email: jens.rantil@tink.se
Phone: +46 708 84 18 32
Web: www.tink.se

Facebook Linkedin Twitter

--
Jens Rantil
Backend engineer
Tink AB

Email: jens.rantil@tink.se
Phone: +46 708 84 18 32
Web: www.tink.se

Facebook Linkedin Twitter

Re: InterProcessSemaphoreMutex not cleaning up old znodes

Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.

This has long been an issue in ZooKeeper. The good news is that there are solutions:

-Jordan

On September 13, 2015 at 11:41:41 AM, Jens Rantil (jens.rantil@tink.se) wrote:

Hi again,

I have a distributed application which has InterProcessSemaphoreMutexes which occasionally are taken per user. To do this, the mutex path is appended with a user's unique ID (example: /mylock/{userid}). All is fine and dandy and this generally works. However, I've noticed that the paths given to the InterProcessSemaphoreMutexes aren't cleaned up after I'm done with the locks.

This means, my Zookeeper ensemble has every single one of my previous mutexes held in memory. When I list my znodes I see:
/mylock/1
/mylock/2
...
/mylock/XXX

Given that I generally only take ~20 locks at a time it feels unnecessary to keep all these in memory. We have had to increase initLimit in Zookeeper configuration, most certainly due to all of these znodes.

--
Jens Rantil
Backend engineer
Tink AB

Email: jens.rantil@tink.se
Phone: +46 708 84 18 32
Web: www.tink.se

Facebook Linkedin Twitter