You are viewing a plain text version of this content. The canonical link for it is here.
Posted to proton@qpid.apache.org by "Darryl L. Pierce" <dp...@redhat.com> on 2015/01/21 20:22:13 UTC
Ruby and the Engine APIs
I've been working on providing the low-level engine APIs in Ruby over
the past few weeks, and from what I can see I think I'm near to
completion regarding wrapping them.
However, I'm not sure if I'm missing something since I don't see a way
to use the code to create a connection. :D
Specifically, what I want to do next is create examples of working with
these low-level APIs, initially a simple send/receive example would be
best with a receiver that listens for a connection and a sender that
connects to it, all while being able to toggle tracing the
communication, etc. I see nothing existing for other languages to use as
a guide, so am a little stumped at the moment.
The work I've done to date is here:
https://github.com/mcpierce/Proton/tree/PROTON-799-Ruby-engine-apis
--
Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc.
Delivering value year after year.
Red Hat ranks #1 in value among software vendors.
http://www.redhat.com/promo/vendor/
Re: Ruby memory management (was: Ruby and the Engine APIs)
Posted by Rafael Schloming <rh...@alum.mit.edu>.
On Fri, Jan 30, 2015 at 10:40 AM, Darryl L. Pierce <dp...@redhat.com>
wrote:
> On Wed, Jan 28, 2015 at 01:22:45PM -0500, Rafael Schloming wrote:
> <snip>
> > Also, have you been able to validate your testing strategy for
> either/both
> > of these POCs? Can you generate seg faults and/or valgrind warnings when
> > you intentionally comment out the line of code that keeps the reference
> > alive?
>
> The POC that uses manual wrapping of a C struct works correctly,
> preventing objects from being reaped without leaking memory.
>
> I validated this by creating exhaustive (1M+) instances of both pure Ruby
> and C structs that have been wrapped via the Data_Wrap_Struct, assigning
> the Ruby object to the C struct so that only C held a reference to it,
> then calling GC.start to reap objects and then checking that the
> expected number of the pure Ruby objects still existed, via
> ObjectSpace.each_object([class]).count. Accessing the C-help Ruby object
> and doing functions such as class_eval on it worked without segmentation
> faults.
>
> I then ensures that it wasn't a fluke by commenting out, in the function
> that marks the Ruby object in C to keep it from being reaped, and
> re-running the tests. The app *immediately* segfaults after trying to
> class_eval the first Ruby object after garbage collection.
>
> So this path is the right one to follow.
>
This sounds promising, is there any way you could highlight the key bits to
look at on the branch you pointed to? Maybe post it in a way that would
permit line by line comments? (My git-fu isn't super strong, so if there's
already a way to do this with what you posted I apologize for the request.)
--Rafael
Re: Ruby memory management (was: Ruby and the Engine APIs)
Posted by "Darryl L. Pierce" <dp...@redhat.com>.
On Wed, Jan 28, 2015 at 01:22:45PM -0500, Rafael Schloming wrote:
<snip>
> Also, have you been able to validate your testing strategy for either/both
> of these POCs? Can you generate seg faults and/or valgrind warnings when
> you intentionally comment out the line of code that keeps the reference
> alive?
The POC that uses manual wrapping of a C struct works correctly,
preventing objects from being reaped without leaking memory.
I validated this by creating exhaustive (1M+) instances of both pure Ruby
and C structs that have been wrapped via the Data_Wrap_Struct, assigning
the Ruby object to the C struct so that only C held a reference to it,
then calling GC.start to reap objects and then checking that the
expected number of the pure Ruby objects still existed, via
ObjectSpace.each_object([class]).count. Accessing the C-help Ruby object
and doing functions such as class_eval on it worked without segmentation
faults.
I then ensures that it wasn't a fluke by commenting out, in the function
that marks the Ruby object in C to keep it from being reaped, and
re-running the tests. The app *immediately* segfaults after trying to
class_eval the first Ruby object after garbage collection.
So this path is the right one to follow.
--
Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc.
Delivering value year after year.
Red Hat ranks #1 in value among software vendors.
http://www.redhat.com/promo/vendor/
Re: Ruby memory management (was: Ruby and the Engine APIs)
Posted by "Darryl L. Pierce" <dp...@redhat.com>.
On Wed, Jan 28, 2015 at 01:10:28PM -0500, Rafael Schloming wrote:
> On Wed, Jan 28, 2015 at 1:05 PM, Darryl L. Pierce <dp...@redhat.com>
> wrote:
>
> > On Wed, Jan 28, 2015 at 12:06:44PM -0500, Rafael Schloming wrote:
> > > Why did you reject it then?
> >
> > Reject it? I don't recall rejecting any option.
>
> I meant why did you post about the global array thing and not this. Is
> there some reason you think one is preferred to the other?
I described it since it was the one different from what we discussed
early based on the blog posts.
--
Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc.
Delivering value year after year.
Red Hat ranks #1 in value among software vendors.
http://www.redhat.com/promo/vendor/
Re: Ruby memory management (was: Ruby and the Engine APIs)
Posted by Rafael Schloming <rh...@alum.mit.edu>.
On Wed, Jan 28, 2015 at 1:05 PM, Darryl L. Pierce <dp...@redhat.com>
wrote:
> On Wed, Jan 28, 2015 at 12:06:44PM -0500, Rafael Schloming wrote:
> > Why did you reject it then?
>
> Reject it? I don't recall rejecting any option.
>
I meant why did you post about the global array thing and not this. Is
there some reason you think one is preferred to the other?
--Rafael
Re: Ruby memory management (was: Ruby and the Engine APIs)
Posted by "Darryl L. Pierce" <dp...@redhat.com>.
On Wed, Jan 28, 2015 at 12:06:44PM -0500, Rafael Schloming wrote:
> Why did you reject it then?
Reject it? I don't recall rejecting any option.
--
Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc.
Delivering value year after year.
Red Hat ranks #1 in value among software vendors.
http://www.redhat.com/promo/vendor/
Re: Ruby memory management (was: Ruby and the Engine APIs)
Posted by Rafael Schloming <rh...@alum.mit.edu>.
No, you posted about a POC involving keeping C references alive by putting
them in a global array. I then commented that it sounded like it could be
brittle and might be more complicated than just wrapping the pn_rubyref_t
struct manually and directly integrating with ruby gc. You then said you
had a POC that did that also. Is there a reason you didn't mention the
latter POC first? Which POC is better in your estimation and why?
Also, have you been able to validate your testing strategy for either/both
of these POCs? Can you generate seg faults and/or valgrind warnings when
you intentionally comment out the line of code that keeps the reference
alive?
--Rafael
On Wed, Jan 28, 2015 at 1:12 PM, Darryl L. Pierce <dp...@redhat.com>
wrote:
> On Wed, Jan 28, 2015 at 12:06:44PM -0500, Rafael Schloming wrote:
> > Why did you reject it then?
>
> Are you referring to this?
>
> "Though, I was hoping we could avoid having to manually do things..."
>
> What I meant was that I would like to keep the work within the confines
> of the Swig code.
>
> --
> Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc.
> Delivering value year after year.
> Red Hat ranks #1 in value among software vendors.
> http://www.redhat.com/promo/vendor/
>
>
Re: Ruby memory management (was: Ruby and the Engine APIs)
Posted by "Darryl L. Pierce" <dp...@redhat.com>.
On Wed, Jan 28, 2015 at 12:06:44PM -0500, Rafael Schloming wrote:
> Why did you reject it then?
Are you referring to this?
"Though, I was hoping we could avoid having to manually do things..."
What I meant was that I would like to keep the work within the confines
of the Swig code.
--
Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc.
Delivering value year after year.
Red Hat ranks #1 in value among software vendors.
http://www.redhat.com/promo/vendor/
Re: Ruby memory management (was: Ruby and the Engine APIs)
Posted by Rafael Schloming <rh...@alum.mit.edu>.
Why did you reject it then?
--Rafael
On Wed, Jan 28, 2015 at 9:54 AM, Darryl L. Pierce <dp...@redhat.com>
wrote:
> On Wed, Jan 28, 2015 at 09:19:29AM -0500, Rafael Schloming wrote:
> > On Wed, Jan 28, 2015 at 9:06 AM, Darryl L. Pierce <dp...@redhat.com>
> > wrote:
> >
> > > On Wed, Jan 28, 2015 at 06:04:57AM -0500, Rafael Schloming wrote:
> > > > On the face of it this sounds like it could be quite brittle and
> probably
> > > > more complicated than just forgetting about swig for the one
> pn_rubyref_t
> > > > struct and wrapping it manually. Did you attempt the latter option at
> > > all?
> > >
> > > Yes, there is a POC of this on my branch as well.
> > >
> >
> > Did it work? Can you send me a pointer to it?
>
> Yes, it works. It uses Data_Wrap_Struct to encapsulate the pn_rubyref_t
> type, rb_gc_mark to mark any Ruby object held by that type against
> reaping, and does appropriate alloc and free operations on instances of
> the struct.
>
> https://github.com/mcpierce/Proton/tree/c-to-ruby-reference-gc-check
>
> --
> Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc.
> Delivering value year after year.
> Red Hat ranks #1 in value among software vendors.
> http://www.redhat.com/promo/vendor/
>
>
Re: Ruby memory management (was: Ruby and the Engine APIs)
Posted by "Darryl L. Pierce" <dp...@redhat.com>.
On Wed, Jan 28, 2015 at 09:19:29AM -0500, Rafael Schloming wrote:
> On Wed, Jan 28, 2015 at 9:06 AM, Darryl L. Pierce <dp...@redhat.com>
> wrote:
>
> > On Wed, Jan 28, 2015 at 06:04:57AM -0500, Rafael Schloming wrote:
> > > On the face of it this sounds like it could be quite brittle and probably
> > > more complicated than just forgetting about swig for the one pn_rubyref_t
> > > struct and wrapping it manually. Did you attempt the latter option at
> > all?
> >
> > Yes, there is a POC of this on my branch as well.
> >
>
> Did it work? Can you send me a pointer to it?
Yes, it works. It uses Data_Wrap_Struct to encapsulate the pn_rubyref_t
type, rb_gc_mark to mark any Ruby object held by that type against
reaping, and does appropriate alloc and free operations on instances of
the struct.
https://github.com/mcpierce/Proton/tree/c-to-ruby-reference-gc-check
--
Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc.
Delivering value year after year.
Red Hat ranks #1 in value among software vendors.
http://www.redhat.com/promo/vendor/
Re: Ruby memory management (was: Ruby and the Engine APIs)
Posted by Rafael Schloming <rh...@alum.mit.edu>.
On Wed, Jan 28, 2015 at 9:06 AM, Darryl L. Pierce <dp...@redhat.com>
wrote:
> On Wed, Jan 28, 2015 at 06:04:57AM -0500, Rafael Schloming wrote:
> > On the face of it this sounds like it could be quite brittle and probably
> > more complicated than just forgetting about swig for the one pn_rubyref_t
> > struct and wrapping it manually. Did you attempt the latter option at
> all?
>
> Yes, there is a POC of this on my branch as well.
>
Did it work? Can you send me a pointer to it?
--Rafael
Re: Ruby memory management (was: Ruby and the Engine APIs)
Posted by "Darryl L. Pierce" <dp...@redhat.com>.
On Wed, Jan 28, 2015 at 06:04:57AM -0500, Rafael Schloming wrote:
> On the face of it this sounds like it could be quite brittle and probably
> more complicated than just forgetting about swig for the one pn_rubyref_t
> struct and wrapping it manually. Did you attempt the latter option at all?
Yes, there is a POC of this on my branch as well.
--
Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc.
Delivering value year after year.
Red Hat ranks #1 in value among software vendors.
http://www.redhat.com/promo/vendor/
Re: Ruby memory management (was: Ruby and the Engine APIs)
Posted by Rafael Schloming <rh...@alum.mit.edu>.
On Tue, Jan 27, 2015 at 5:35 PM, Darryl L. Pierce <dp...@redhat.com>
wrote:
> On Fri, Jan 23, 2015 at 03:46:34PM -0500, Darryl L. Pierce wrote:
> > +1 Though, I was hoping we could avoid having to manually do things...
>
> So I have a working POC that assigns a Ruby object to a C struct in such
> a way as to keep the Ruby object from being reaped. The solution (for
> now) stores the object in a hidden global array for such objects for as
> long as they're held by the C structure and, when C is deleted or the
> reference changed, the object is removed from the array and available
> for reaping.
>
> I submitted a question to the Swig users mailing list, but that seems to
> be pretty low traffic and effectively unmanned ATM. Only 15 posts there
> in the last month and none of them have followups.
>
On the face of it this sounds like it could be quite brittle and probably
more complicated than just forgetting about swig for the one pn_rubyref_t
struct and wrapping it manually. Did you attempt the latter option at all?
It's really important to get this part right. If this isn't done well, then
the whole binding will be unstable. Whatever has been done here with the
current ruby bindings seems to seg fault about once every 10 test runs or
so. We really can't afford to repeat that.
--Rafael
Re: Ruby memory management (was: Ruby and the Engine APIs)
Posted by "Darryl L. Pierce" <dp...@redhat.com>.
On Fri, Jan 23, 2015 at 03:46:34PM -0500, Darryl L. Pierce wrote:
> +1 Though, I was hoping we could avoid having to manually do things...
So I have a working POC that assigns a Ruby object to a C struct in such
a way as to keep the Ruby object from being reaped. The solution (for
now) stores the object in a hidden global array for such objects for as
long as they're held by the C structure and, when C is deleted or the
reference changed, the object is removed from the array and available
for reaping.
I submitted a question to the Swig users mailing list, but that seems to
be pretty low traffic and effectively unmanned ATM. Only 15 posts there
in the last month and none of them have followups.
--
Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc.
Delivering value year after year.
Red Hat ranks #1 in value among software vendors.
http://www.redhat.com/promo/vendor/
Re: Ruby memory management (was: Ruby and the Engine APIs)
Posted by "Darryl L. Pierce" <dp...@redhat.com>.
On Fri, Jan 23, 2015 at 03:25:33PM -0500, Rafael Schloming wrote:
> On Fri, Jan 23, 2015 at 2:08 PM, Darryl L. Pierce <dp...@redhat.com>
> wrote:
>
> > On Fri, Jan 23, 2015 at 01:49:34PM -0500, Rafael Schloming wrote:
> > > It does talk about what swig does, but it also talks about the other
> > > direction:
> > >
> > > "If the C structure references other ruby objects, then the mark function
> > > pointer must also be provided and must properly mark the other objects
> > with
> > > rb_gc_mark()"
> >
> > What I meant is it's not showing how to do that. Swig is the one that's
> > generating that Ruby struct wrapping, but I haven't found (yet) if they
> > expose to users a way to tap into that adn do what we want.
> >
>
> If you can find a way for swig to let you control it that's great, but I
> was thinking we could just wrap this one struct by hand. It shouldn't be a
> whole lot of code for just that struct, and swig can still handle the rest
> of them.
+1 Though, I was hoping we could avoid having to manually do things...
--
Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc.
Delivering value year after year.
Red Hat ranks #1 in value among software vendors.
http://www.redhat.com/promo/vendor/
Re: Ruby memory management (was: Ruby and the Engine APIs)
Posted by Rafael Schloming <rh...@alum.mit.edu>.
On Fri, Jan 23, 2015 at 2:08 PM, Darryl L. Pierce <dp...@redhat.com>
wrote:
> On Fri, Jan 23, 2015 at 01:49:34PM -0500, Rafael Schloming wrote:
> > It does talk about what swig does, but it also talks about the other
> > direction:
> >
> > "If the C structure references other ruby objects, then the mark function
> > pointer must also be provided and must properly mark the other objects
> with
> > rb_gc_mark()"
>
> What I meant is it's not showing how to do that. Swig is the one that's
> generating that Ruby struct wrapping, but I haven't found (yet) if they
> expose to users a way to tap into that adn do what we want.
>
If you can find a way for swig to let you control it that's great, but I
was thinking we could just wrap this one struct by hand. It shouldn't be a
whole lot of code for just that struct, and swig can still handle the rest
of them.
--Rafael
Re: Ruby memory management (was: Ruby and the Engine APIs)
Posted by "Darryl L. Pierce" <dp...@redhat.com>.
On Fri, Jan 23, 2015 at 01:49:34PM -0500, Rafael Schloming wrote:
> It does talk about what swig does, but it also talks about the other
> direction:
>
> "If the C structure references other ruby objects, then the mark function
> pointer must also be provided and must properly mark the other objects with
> rb_gc_mark()"
What I meant is it's not showing how to do that. Swig is the one that's
generating that Ruby struct wrapping, but I haven't found (yet) if they
expose to users a way to tap into that adn do what we want.
--
Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc.
Delivering value year after year.
Red Hat ranks #1 in value among software vendors.
http://www.redhat.com/promo/vendor/
Re: Ruby memory management (was: Ruby and the Engine APIs)
Posted by Rafael Schloming <rh...@alum.mit.edu>.
It does talk about what swig does, but it also talks about the other
direction:
"If the C structure references other ruby objects, then the mark function
pointer must also be provided and must properly mark the other objects with
rb_gc_mark()"
--Rafael
On Fri, Jan 23, 2015 at 1:24 PM, Darryl L. Pierce <dp...@redhat.com>
wrote:
> On Fri, Jan 23, 2015 at 11:02:59AM -0500, Rafael Schloming wrote:
> <snip>
> > For more info on how to integrate with Ruby's GC you can read this
> > article[1]. It's one of the few pieces of documentation I've found that
> > actually explain how to keep a reference from C to a Ruby object.
> >
> > [1]
> >
> http://clalance.blogspot.com/2013/11/writing-ruby-extensions-in-c-part-13.html
>
> This blog post shows how to manually do what Swig is doing for us:
> represent a C struct as something Ruby can touch, with hooks to release
> memory when GC runs on the Ruby wrapper. But, sadly, it's not showing
> what we need, which is how to assign a reference to a Ruby object to a C
> struct and then have Ruby not GC that Ruby object.
>
> --
> Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc.
> Delivering value year after year.
> Red Hat ranks #1 in value among software vendors.
> http://www.redhat.com/promo/vendor/
>
>
Re: Ruby memory management (was: Ruby and the Engine APIs)
Posted by "Darryl L. Pierce" <dp...@redhat.com>.
On Fri, Jan 23, 2015 at 11:02:59AM -0500, Rafael Schloming wrote:
<snip>
> For more info on how to integrate with Ruby's GC you can read this
> article[1]. It's one of the few pieces of documentation I've found that
> actually explain how to keep a reference from C to a Ruby object.
>
> [1]
> http://clalance.blogspot.com/2013/11/writing-ruby-extensions-in-c-part-13.html
This blog post shows how to manually do what Swig is doing for us:
represent a C struct as something Ruby can touch, with hooks to release
memory when GC runs on the Ruby wrapper. But, sadly, it's not showing
what we need, which is how to assign a reference to a Ruby object to a C
struct and then have Ruby not GC that Ruby object.
--
Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc.
Delivering value year after year.
Red Hat ranks #1 in value among software vendors.
http://www.redhat.com/promo/vendor/
Re: Ruby memory management (was: Ruby and the Engine APIs)
Posted by Rafael Schloming <rh...@alum.mit.edu>.
That sounds like progress, but from what you're describing I'm not sure
you're actually testing C holding onto a reference to Ruby. As you say,
swig is helping you out with the Ruby -> C direction, but we need to be
able to make that void * reference actually point to a ruby object (that is
not pointed to by any other ruby object) and still keep it alive. If you
haven't had to integrate with the ruby GC system yet then there is a pretty
good chance you aren't actually testing this. It can be a bit tricky to
test for a couple reasons since a) you need to force the GC to run, and b)
you need to ensure there are no other references to the ruby object that is
being pointed to by the C object.
In fact now that I think about it, you really probably want a negative test
to ensure that your testing strategy is working, i.e. keep tweaking your
test until you get valgrind warnings telling you that you're accessing
freed memory. Then enable the GC integration, and verify that those
valgrind warnings go away.
For more info on how to integrate with Ruby's GC you can read this
article[1]. It's one of the few pieces of documentation I've found that
actually explain how to keep a reference from C to a Ruby object.
[1]
http://clalance.blogspot.com/2013/11/writing-ruby-extensions-in-c-part-13.html
--Rafael
On Fri, Jan 23, 2015 at 9:01 AM, Darryl L. Pierce <dp...@redhat.com>
wrote:
> On Thu, Jan 22, 2015 at 12:08:52PM -0500, Rafael Schloming wrote:
> > The most important thing to get worked out for this is the memory
> > management semantics between C and Ruby. From what I can tell from your
> > branch, it looks like you haven't done that yet.
> <snip>
>
> My initial readings into how Ruby handles object references initially
> lead me to believe that, with Swig, we're receiving some benefit
> already WRT object management. Since Ruby uses mark and sweep rather
> than reference counts, when a Ruby object (or an C object wrapped by
> Swig as a Ruby object) goes out of reference then it'll get garbage
> collected. But I won't lie: there might be some subtle detail I'm
> missing here.
>
> I played around with this last night after your suggestion and wrote
> some stuff on a side branch [1]. It's a simple test that uses the
> following types:
>
> * pn_rubyref_t - C type in Proton that's wrapped by Swig; holds a void *
> reference to any kind of object
> * Farkle - a pure Ruby class
>
> I then wrote a simple app that creates 1M instances of rb_rubyref_t and
> 1M instances of Farkle as assigns a Farkle to each rb_rubyref_t. It then
> puts them each into an array, which is size limited to 100k entries (to
> allow any GC to run while the app is going).
>
> I also added finalizer function to Farkle that just outputs some text when
> the object is being garbage collected.
>
> What I see as the app runs is the output of the Farkle instances being
> garbage collected while there are still instances being created.
>
> The app then ends by sleeping and then exiting.
>
> Before and after sleeping the app outputs the number of objects (via
> ObjectSpace) exist for each type. Running it multiple times I see all of
> the pn_rubyref_t and Farkle instances being cleaned up, no memory
> leaks.
>
> I then changed things to make a circular reference between Farkle and
> pn_rubyref_t. Re-ran the tests and still see the objects getting cleaned
> up. I also ran top to keep an eye on memory usage for the ruby-mri
> process.
>
> O_o (1) [J:0/1028] mcpierce@mcpierce-laptop:cmake (0.3-fedora) $ top -b |
> grep ruby-mri
> 19989 mcpierce 20 0 188964 23228 6380 S 53.3 0.3 0:00.55
> ruby-mri
> 19989 mcpierce 20 0 202644 36892 6380 S 18.3 0.5 0:01.10
> ruby-mri
> 19989 mcpierce 20 0 207492 41784 6380 R 19.9 0.5 0:01.70
> ruby-mri
> 19989 mcpierce 20 0 227748 62108 6380 S 17.6 0.8 0:02.23
> ruby-mri
> 19989 mcpierce 20 0 229796 64020 6380 R 22.3 0.8 0:02.90
> ruby-mri
> 19989 mcpierce 20 0 233096 67188 6380 R 57.1 0.8 0:03.42
> ruby-mri
> 19989 mcpierce 20 0 233096 67188 6380 S 0.7 0.8 0:03.44
> ruby-mri
> 19989 mcpierce 20 0 233096 67188 6380 S 0.0 0.8 0:03.44
> ruby-mri
> 19989 mcpierce 20 0 233096 67188 6380 S 0.0 0.8 0:03.44
> ruby-mri
> 19989 mcpierce 20 0 233096 67188 6380 S 1.3 0.8 0:03.48
> ruby-mri
> 19989 mcpierce 20 0 235340 69564 6380 R 48.5 0.9 0:04.94
> ruby-mri
> 19989 mcpierce 20 0 235340 69564 6380 S 47.5 0.9 0:06.37
> ruby-mri
> 19989 mcpierce 20 0 235340 69564 6380 R 50.5 0.9 0:07.89
> ruby-mri
> 19989 mcpierce 20 0 235340 69564 6380 S 48.0 0.9 0:09.34
> ruby-mri
>
> I only show those lines since, after that point, the virtual memory
> footprint didn't grow for the run os the apps run.
>
>
> [1] https://github.com/mcpierce/Proton/tree/c-to-ruby-reference-gc-check
>
> --
> Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc.
> Delivering value year after year.
> Red Hat ranks #1 in value among software vendors.
> http://www.redhat.com/promo/vendor/
>
>
Ruby memory management (was: Ruby and the Engine APIs)
Posted by "Darryl L. Pierce" <dp...@redhat.com>.
On Thu, Jan 22, 2015 at 12:08:52PM -0500, Rafael Schloming wrote:
> The most important thing to get worked out for this is the memory
> management semantics between C and Ruby. From what I can tell from your
> branch, it looks like you haven't done that yet.
<snip>
My initial readings into how Ruby handles object references initially
lead me to believe that, with Swig, we're receiving some benefit
already WRT object management. Since Ruby uses mark and sweep rather
than reference counts, when a Ruby object (or an C object wrapped by
Swig as a Ruby object) goes out of reference then it'll get garbage
collected. But I won't lie: there might be some subtle detail I'm
missing here.
I played around with this last night after your suggestion and wrote
some stuff on a side branch [1]. It's a simple test that uses the
following types:
* pn_rubyref_t - C type in Proton that's wrapped by Swig; holds a void *
reference to any kind of object
* Farkle - a pure Ruby class
I then wrote a simple app that creates 1M instances of rb_rubyref_t and
1M instances of Farkle as assigns a Farkle to each rb_rubyref_t. It then
puts them each into an array, which is size limited to 100k entries (to
allow any GC to run while the app is going).
I also added finalizer function to Farkle that just outputs some text when
the object is being garbage collected.
What I see as the app runs is the output of the Farkle instances being
garbage collected while there are still instances being created.
The app then ends by sleeping and then exiting.
Before and after sleeping the app outputs the number of objects (via
ObjectSpace) exist for each type. Running it multiple times I see all of
the pn_rubyref_t and Farkle instances being cleaned up, no memory
leaks.
I then changed things to make a circular reference between Farkle and
pn_rubyref_t. Re-ran the tests and still see the objects getting cleaned
up. I also ran top to keep an eye on memory usage for the ruby-mri
process.
O_o (1) [J:0/1028] mcpierce@mcpierce-laptop:cmake (0.3-fedora) $ top -b | grep ruby-mri
19989 mcpierce 20 0 188964 23228 6380 S 53.3 0.3 0:00.55 ruby-mri
19989 mcpierce 20 0 202644 36892 6380 S 18.3 0.5 0:01.10 ruby-mri
19989 mcpierce 20 0 207492 41784 6380 R 19.9 0.5 0:01.70 ruby-mri
19989 mcpierce 20 0 227748 62108 6380 S 17.6 0.8 0:02.23 ruby-mri
19989 mcpierce 20 0 229796 64020 6380 R 22.3 0.8 0:02.90 ruby-mri
19989 mcpierce 20 0 233096 67188 6380 R 57.1 0.8 0:03.42 ruby-mri
19989 mcpierce 20 0 233096 67188 6380 S 0.7 0.8 0:03.44 ruby-mri
19989 mcpierce 20 0 233096 67188 6380 S 0.0 0.8 0:03.44 ruby-mri
19989 mcpierce 20 0 233096 67188 6380 S 0.0 0.8 0:03.44 ruby-mri
19989 mcpierce 20 0 233096 67188 6380 S 1.3 0.8 0:03.48 ruby-mri
19989 mcpierce 20 0 235340 69564 6380 R 48.5 0.9 0:04.94 ruby-mri
19989 mcpierce 20 0 235340 69564 6380 S 47.5 0.9 0:06.37 ruby-mri
19989 mcpierce 20 0 235340 69564 6380 R 50.5 0.9 0:07.89 ruby-mri
19989 mcpierce 20 0 235340 69564 6380 S 48.0 0.9 0:09.34 ruby-mri
I only show those lines since, after that point, the virtual memory
footprint didn't grow for the run os the apps run.
[1] https://github.com/mcpierce/Proton/tree/c-to-ruby-reference-gc-check
--
Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc.
Delivering value year after year.
Red Hat ranks #1 in value among software vendors.
http://www.redhat.com/promo/vendor/
Re: Ruby and the Engine APIs
Posted by Rafael Schloming <rh...@alum.mit.edu>.
The most important thing to get worked out for this is the memory
management semantics between C and Ruby. From what I can tell from your
branch, it looks like you haven't done that yet.
As I've said before, the first step to wrapping the engine API is to work
out a simple strategy for wrapping C objects from the host language. In the
case of python this is done with the wrapper.py mixin. It uses various
python hooks as well as elements of python's C extension API to allow C
objects to hold references to python objects and keep them alive according
to python's memory management semantics. Just translating this wrapper code
from python to ruby won't do much good as Ruby's memory management
semantics are different and it's C extension API is different.
The first step you should take before bothering to wrap any code is to
figure out how you can have C objects hold pointers to ruby objects without
either incurring memory leaks or seg faults. Judging from what you've done
in ruby.i:
+ static void pn_rbref_incref(void *object) { + // no reference counting
in Ruby + } + + static void pn_rbref_decref(void *object) { + // no
reference counting in Ruby + } + + static int pn_rbref_refcount(void
*object) { + return 1; + } + + static const pn_class_t
*pn_rbref_reify(void *object) { + return PN_RBREF; + }
I suspect if you were to actually try anything significant you would incur
either seg faults or memory leaks or both, depending on precisely how Ruby
GC works. The excerpt from your branch above is creating a class of pointer
that will lie to proton and tell it that it always has a reference when it
doesn't really, and yet also lie to the Ruby VM because it is keeping a
pointer to a Ruby object without any way for the Ruby GC mechanism to be
informed.
I recommend starting with a very simple test case for one object, i.e.
define a wrapper mixin for ruby and use it to wrap just one proton object,
e.g. Connection. Don't bother with wrapping any of the methods on
Connection, all you really want to do is verify that object
creation/deletion works properly. Once you've done that you can run
creation/deletion in a loop and verify that you don't leak memory. Before
this though, you'll need to read up on Ruby's C extension API in order to
figure out how to interface to it's memory management system so that the C
objects can hold pointers to Ruby objects and actually keep them alive, and
also free them when the C objects go away. Until you can do this and get
the basic loop test working with a simple object, there isn't really much
point in trying to wrap anything else.
--Rafael
On Wed, Jan 21, 2015 at 2:22 PM, Darryl L. Pierce <dp...@redhat.com>
wrote:
> I've been working on providing the low-level engine APIs in Ruby over
> the past few weeks, and from what I can see I think I'm near to
> completion regarding wrapping them.
>
> However, I'm not sure if I'm missing something since I don't see a way
> to use the code to create a connection. :D
>
> Specifically, what I want to do next is create examples of working with
> these low-level APIs, initially a simple send/receive example would be
> best with a receiver that listens for a connection and a sender that
> connects to it, all while being able to toggle tracing the
> communication, etc. I see nothing existing for other languages to use as
> a guide, so am a little stumped at the moment.
>
> The work I've done to date is here:
>
> https://github.com/mcpierce/Proton/tree/PROTON-799-Ruby-engine-apis
>
> --
> Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc.
> Delivering value year after year.
> Red Hat ranks #1 in value among software vendors.
> http://www.redhat.com/promo/vendor/
>
>