LDAP and WeakReferences

[sebb <se...@gmail.com>]

I'm not sure what is the purpose of the weak references when applied
to instances of the LDAP class.

Why does an instance not use hard references so that they last for the
life of the instance?

Re: LDAP and WeakReferences

[Matt Sicker <bo...@gmail.com>]

I can’t say my knowledge about Ruby is even on the same planet as Sam’s,
though this pattern reminds me of comparable issues in the Java realm with
ClassLoader leaks in servlets. Sam’s explanation makes sense to me in that
analogue.

On Sat, Jan 18, 2020 at 16:28 Sam Ruby <ru...@intertwingly.net> wrote:

> On Sat, Jan 18, 2020 at 4:56 PM sebb <se...@gmail.com> wrote:
> >
> > On Sat, 18 Jan 2020 at 14:15, Sam Ruby <ru...@intertwingly.net> wrote:
> > >
> > > On Sat, Jan 18, 2020 at 7:48 AM sebb <se...@gmail.com> wrote:
> > > >
> > > > On Sat, 18 Jan 2020 at 12:39, Sam Ruby <ru...@intertwingly.net>
> wrote:
> > > > >
> > > > > On Fri, Jan 17, 2020 at 5:10 PM sebb <se...@gmail.com> wrote:
> > > > > >
> > > > > > I'm not sure what is the purpose of the weak references when
> applied
> > > > > > to instances of the LDAP class.
> > > > > >
> > > > > > Why does an instance not use hard references so that they last
> for the
> > > > > > life of the instance?
> > > > >
> > > > > Passenger applications like the roster tool, the secretary
> workbench,
> > > > > and board agenda tool can stay up for hours or even days.  See:
> > > > > https://whimsy.apache.org/status/passenger
> > > >
> > > > I see.
> > > >
> > > > > After every request, every hard reference will go away, enabling
> the
> > > > > object to be garbage collected.
> > > >
> > > > I don't follow.
> > > >
> > > > If the hard references go away, then surely the objects can be
> > > > garbage-collected?
> > >
> > > ASF::LDAP is a module.  It only goes away when the process exits.
> > > Classes that inherit from ASF::LDAP base have a class instance
> > > variable which contain a collection of instances.
> >
> > Are you referring the @collection class variable?
>
> Yes
>
> > If so, then AFAICT the entries are weak references, so don't affect GC.
>
> Correct, but GC can affect the objects in the collection.  If you read
> ahead (to the paragraph that you indicated that you understand), all
> it takes is for a CGI script or a request to have a hard reference to
> the same data to ensure that objects in the collection is not freed
> until the script/request is complete.
>
> > > Class instance variables go away when the process exits.
> >
> > Ok, but I still don't understand what you meant by:
> >
> > "After every request, every hard reference will go away,"
>
> With Sinatra, a request is handled by a block.  Something like the
> following:
>
> get '/' do
>   ...
> end
>
> Inside that block, a number of hard references may be created.  If
> done inside this block, this would typically be to store something in
> an instance variable.  If done inside a method this is called directly
> or indirectly, it would be stored in a local variable.  Either way,
> all hard references would be release by the end of the request.
>
> What is important is that no class or module instance variable contain
> a hard reference.
>
> > > A common pattern in both CGI scripts and passenger application within
> > > Whimsy is to fetch everything you might need, ideally within one LDAP
> > > request.  Maintain a hard reference on that collection, but don't
> > > necessarily pass those objections around the application.  Methods
> > > that you call that seek to find instances of those collections will
> > > resolve to the same instance, by that I mean that
> > > ASF::Person.find('rubys') will always return the same object, not
> > > merely another object with the same values for attributes.  So if I've
> > > already fetched, for example, public_name, it will be returned without
> > > another round trip to LDAP.
> >
> > That paragraph I understand.
>
> That paragraph is what I was trying to say with the previous sentences.
>
> Rephrasing: if, after every request, there are no hard references left
> to any LDAP objects, all LDAP collections are eligible for garbage
> collection.  My experience is that the Ruby runtime is pretty
> aggressive about garbage collection, so all of the heap activity that
> normally accompanies the completion of a request triggers garbage
> collection.  But, in the case of the secretary workbench, I invested
> some time to add some guarantees:
> https://github.com/apache/whimsy/blob/master/lib/whimsy/asf/rack.rb#L87.
> Since that has been running without a problem for literally years, it
> should be safe to add that line to the config.ru files to the board
> agenda and roster tools.
>
> > > > > What this generally means is that a
> > > > > request will have a consistent view of LDAP, and new requests will
> get
> > > > > fresh data.
> > > >
> > > > Understood.
> > > >
> > > > > - Sam Ruby
> > >
> > > - Sam Ruby
>
> - Sam Ruby
>
-- 
Matt Sicker <bo...@gmail.com>

Re: LDAP and WeakReferences

[Sam Ruby <ru...@intertwingly.net>]

On Sat, Jan 18, 2020 at 4:56 PM sebb <se...@gmail.com> wrote:
>
> On Sat, 18 Jan 2020 at 14:15, Sam Ruby <ru...@intertwingly.net> wrote:
> >
> > On Sat, Jan 18, 2020 at 7:48 AM sebb <se...@gmail.com> wrote:
> > >
> > > On Sat, 18 Jan 2020 at 12:39, Sam Ruby <ru...@intertwingly.net> wrote:
> > > >
> > > > On Fri, Jan 17, 2020 at 5:10 PM sebb <se...@gmail.com> wrote:
> > > > >
> > > > > I'm not sure what is the purpose of the weak references when applied
> > > > > to instances of the LDAP class.
> > > > >
> > > > > Why does an instance not use hard references so that they last for the
> > > > > life of the instance?
> > > >
> > > > Passenger applications like the roster tool, the secretary workbench,
> > > > and board agenda tool can stay up for hours or even days.  See:
> > > > https://whimsy.apache.org/status/passenger
> > >
> > > I see.
> > >
> > > > After every request, every hard reference will go away, enabling the
> > > > object to be garbage collected.
> > >
> > > I don't follow.
> > >
> > > If the hard references go away, then surely the objects can be
> > > garbage-collected?
> >
> > ASF::LDAP is a module.  It only goes away when the process exits.
> > Classes that inherit from ASF::LDAP base have a class instance
> > variable which contain a collection of instances.
>
> Are you referring the @collection class variable?

Yes

> If so, then AFAICT the entries are weak references, so don't affect GC.

Correct, but GC can affect the objects in the collection.  If you read
ahead (to the paragraph that you indicated that you understand), all
it takes is for a CGI script or a request to have a hard reference to
the same data to ensure that objects in the collection is not freed
until the script/request is complete.

> > Class instance variables go away when the process exits.
>
> Ok, but I still don't understand what you meant by:
>
> "After every request, every hard reference will go away,"

With Sinatra, a request is handled by a block.  Something like the following:

get '/' do
  ...
end

Inside that block, a number of hard references may be created.  If
done inside this block, this would typically be to store something in
an instance variable.  If done inside a method this is called directly
or indirectly, it would be stored in a local variable.  Either way,
all hard references would be release by the end of the request.

What is important is that no class or module instance variable contain
a hard reference.

> > A common pattern in both CGI scripts and passenger application within
> > Whimsy is to fetch everything you might need, ideally within one LDAP
> > request.  Maintain a hard reference on that collection, but don't
> > necessarily pass those objections around the application.  Methods
> > that you call that seek to find instances of those collections will
> > resolve to the same instance, by that I mean that
> > ASF::Person.find('rubys') will always return the same object, not
> > merely another object with the same values for attributes.  So if I've
> > already fetched, for example, public_name, it will be returned without
> > another round trip to LDAP.
>
> That paragraph I understand.

That paragraph is what I was trying to say with the previous sentences.

Rephrasing: if, after every request, there are no hard references left
to any LDAP objects, all LDAP collections are eligible for garbage
collection.  My experience is that the Ruby runtime is pretty
aggressive about garbage collection, so all of the heap activity that
normally accompanies the completion of a request triggers garbage
collection.  But, in the case of the secretary workbench, I invested
some time to add some guarantees:
https://github.com/apache/whimsy/blob/master/lib/whimsy/asf/rack.rb#L87.
Since that has been running without a problem for literally years, it
should be safe to add that line to the config.ru files to the board
agenda and roster tools.

> > > > What this generally means is that a
> > > > request will have a consistent view of LDAP, and new requests will get
> > > > fresh data.
> > >
> > > Understood.
> > >
> > > > - Sam Ruby
> >
> > - Sam Ruby

- Sam Ruby

Re: LDAP and WeakReferences

[sebb <se...@gmail.com>]

On Sat, 18 Jan 2020 at 14:15, Sam Ruby <ru...@intertwingly.net> wrote:
>
> On Sat, Jan 18, 2020 at 7:48 AM sebb <se...@gmail.com> wrote:
> >
> > On Sat, 18 Jan 2020 at 12:39, Sam Ruby <ru...@intertwingly.net> wrote:
> > >
> > > On Fri, Jan 17, 2020 at 5:10 PM sebb <se...@gmail.com> wrote:
> > > >
> > > > I'm not sure what is the purpose of the weak references when applied
> > > > to instances of the LDAP class.
> > > >
> > > > Why does an instance not use hard references so that they last for the
> > > > life of the instance?
> > >
> > > Passenger applications like the roster tool, the secretary workbench,
> > > and board agenda tool can stay up for hours or even days.  See:
> > > https://whimsy.apache.org/status/passenger
> >
> > I see.
> >
> > > After every request, every hard reference will go away, enabling the
> > > object to be garbage collected.
> >
> > I don't follow.
> >
> > If the hard references go away, then surely the objects can be
> > garbage-collected?
>
> ASF::LDAP is a module.  It only goes away when the process exits.
> Classes that inherit from ASF::LDAP base have a class instance
> variable which contain a collection of instances.

Are you referring the @collection class variable?
If so, then AFAICT the entries are weak references, so don't affect GC.

> Class instance variables go away when the process exits.

Ok, but I still don't understand what you meant by:

"After every request, every hard reference will go away,"

> A common pattern in both CGI scripts and passenger application within
> Whimsy is to fetch everything you might need, ideally within one LDAP
> request.  Maintain a hard reference on that collection, but don't
> necessarily pass those objections around the application.  Methods
> that you call that seek to find instances of those collections will
> resolve to the same instance, by that I mean that
> ASF::Person.find('rubys') will always return the same object, not
> merely another object with the same values for attributes.  So if I've
> already fetched, for example, public_name, it will be returned without
> another round trip to LDAP.

That paragraph I understand.

> > > What this generally means is that a
> > > request will have a consistent view of LDAP, and new requests will get
> > > fresh data.
> >
> > Understood.
> >
> > > - Sam Ruby
>
> - Sam Ruby

Re: LDAP and WeakReferences

[Sam Ruby <ru...@intertwingly.net>]

On Sat, Jan 18, 2020 at 7:48 AM sebb <se...@gmail.com> wrote:
>
> On Sat, 18 Jan 2020 at 12:39, Sam Ruby <ru...@intertwingly.net> wrote:
> >
> > On Fri, Jan 17, 2020 at 5:10 PM sebb <se...@gmail.com> wrote:
> > >
> > > I'm not sure what is the purpose of the weak references when applied
> > > to instances of the LDAP class.
> > >
> > > Why does an instance not use hard references so that they last for the
> > > life of the instance?
> >
> > Passenger applications like the roster tool, the secretary workbench,
> > and board agenda tool can stay up for hours or even days.  See:
> > https://whimsy.apache.org/status/passenger
>
> I see.
>
> > After every request, every hard reference will go away, enabling the
> > object to be garbage collected.
>
> I don't follow.
>
> If the hard references go away, then surely the objects can be
> garbage-collected?

ASF::LDAP is a module.  It only goes away when the process exits.
Classes that inherit from ASF::LDAP base have a class instance
variable which contain a collection of instances.  Class instance
variables go away when the process exits.

A common pattern in both CGI scripts and passenger application within
Whimsy is to fetch everything you might need, ideally within one LDAP
request.  Maintain a hard reference on that collection, but don't
necessarily pass those objections around the application.  Methods
that you call that seek to find instances of those collections will
resolve to the same instance, by that I mean that
ASF::Person.find('rubys') will always return the same object, not
merely another object with the same values for attributes.  So if I've
already fetched, for example, public_name, it will be returned without
another round trip to LDAP.

> > What this generally means is that a
> > request will have a consistent view of LDAP, and new requests will get
> > fresh data.
>
> Understood.
>
> > - Sam Ruby

- Sam Ruby

Re: LDAP and WeakReferences

[sebb <se...@gmail.com>]

On Sat, 18 Jan 2020 at 12:39, Sam Ruby <ru...@intertwingly.net> wrote:
>
> On Fri, Jan 17, 2020 at 5:10 PM sebb <se...@gmail.com> wrote:
> >
> > I'm not sure what is the purpose of the weak references when applied
> > to instances of the LDAP class.
> >
> > Why does an instance not use hard references so that they last for the
> > life of the instance?
>
> Passenger applications like the roster tool, the secretary workbench,
> and board agenda tool can stay up for hours or even days.  See:
> https://whimsy.apache.org/status/passenger

I see.

> After every request, every hard reference will go away, enabling the
> object to be garbage collected.

I don't follow.

If the hard references go away, then surely the objects can be
garbage-collected?

> What this generally means is that a
> request will have a consistent view of LDAP, and new requests will get
> fresh data.

Understood.

> - Sam Ruby

Re: LDAP and WeakReferences

[Sam Ruby <ru...@intertwingly.net>]

On Fri, Jan 17, 2020 at 5:10 PM sebb <se...@gmail.com> wrote:
>
> I'm not sure what is the purpose of the weak references when applied
> to instances of the LDAP class.
>
> Why does an instance not use hard references so that they last for the
> life of the instance?

Passenger applications like the roster tool, the secretary workbench,
and board agenda tool can stay up for hours or even days.  See:
https://whimsy.apache.org/status/passenger

After every request, every hard reference will go away, enabling the
object to be garbage collected.  What this generally means is that a
request will have a consistent view of LDAP, and new requests will get
fresh data.

- Sam Ruby