You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Jim Jagielski <ji...@jaguNET.com> on 2018/09/26 15:09:00 UTC
Using APR pools "better"
At ApacheCon's welcoming event last night, Greg, Sander and I were chatting and Greg reminded us that the Subversion project "learned a lot about using APR pools" and it seems to me that a lot of that knowledge is likely missing in httpd-land. I also know that some of that has been backported back to APR itself, but I am wondering (as I am wont to do), if we couldn't do a better job here. I am sure that there are things in svn that could be easily and readily folded back into APR w/o breaking anything.
I know that some of that influenced Greg's PoCore work, but it would be really cool if someone in Subversion could maybe point out some of those improvements and "lessons learned" so that both APR and httpd could benefit.
Thx!
Re: Using APR pools "better"
Posted by Stefan Sperling <st...@apache.org>.
On Wed, Sep 26, 2018 at 04:15:19PM -0500, Greg Stein wrote:
> iterpool, scratch_pool, and result_pool are the KEY three concepts that we
> learned while working on Subversion.
Here's a recent example of where and why we added an iterpool (which
should have been added when this loop was written in the first place):
Log message: http://svn.apache.org/viewvc?view=revision&revision=1841725
Diff: http://svn.apache.org/viewvc/subversion/trunk/subversion/svn/conflict-callbacks.c?r1=1841725&r2=1841724&pathrev=1841725
There is a key difference between Subversion and HTTP which affects
pool lifetime concerns: SVN (on the client side) is a blocking
application, while HTTPD is event-driven.
Pools in HTTPD are often bound to the lifetime of a HTTP request (event)
which means they are short-lived by design. Global data stored in pools
is of a relatively fixed size.
Whereas Subversion's client library is often used in a long-running
process context (such the Windows Explorer in case of TortoiseSVN).
The svn command line client alternates between blocking for user input
and calling into the client library. The client library will return results
to these API users and it has no idea about the lifetimes its callers require.
Which is why letting the caller manage pool lifetime is a natural thing
to do for this application.
If you are going to write some general pool usage recommendations for APR
I suppose such docs should mention how the various pool usage models relate
to the software architectures they were invented for.
Re: Using APR pools "better"
Posted by Stefan Sperling <st...@apache.org>.
On Wed, Sep 26, 2018 at 04:15:19PM -0500, Greg Stein wrote:
> iterpool, scratch_pool, and result_pool are the KEY three concepts that we
> learned while working on Subversion.
Here's a recent example of where and why we added an iterpool (which
should have been added when this loop was written in the first place):
Log message: http://svn.apache.org/viewvc?view=revision&revision=1841725
Diff: http://svn.apache.org/viewvc/subversion/trunk/subversion/svn/conflict-callbacks.c?r1=1841725&r2=1841724&pathrev=1841725
There is a key difference between Subversion and HTTP which affects
pool lifetime concerns: SVN (on the client side) is a blocking
application, while HTTPD is event-driven.
Pools in HTTPD are often bound to the lifetime of a HTTP request (event)
which means they are short-lived by design. Global data stored in pools
is of a relatively fixed size.
Whereas Subversion's client library is often used in a long-running
process context (such the Windows Explorer in case of TortoiseSVN).
The svn command line client alternates between blocking for user input
and calling into the client library. The client library will return results
to these API users and it has no idea about the lifetimes its callers require.
Which is why letting the caller manage pool lifetime is a natural thing
to do for this application.
If you are going to write some general pool usage recommendations for APR
I suppose such docs should mention how the various pool usage models relate
to the software architectures they were invented for.
Re: Using APR pools "better"
Posted by Stefan Sperling <st...@apache.org>.
On Wed, Sep 26, 2018 at 04:15:19PM -0500, Greg Stein wrote:
> iterpool, scratch_pool, and result_pool are the KEY three concepts that we
> learned while working on Subversion.
Here's a recent example of where and why we added an iterpool (which
should have been added when this loop was written in the first place):
Log message: http://svn.apache.org/viewvc?view=revision&revision=1841725
Diff: http://svn.apache.org/viewvc/subversion/trunk/subversion/svn/conflict-callbacks.c?r1=1841725&r2=1841724&pathrev=1841725
There is a key difference between Subversion and HTTP which affects
pool lifetime concerns: SVN (on the client side) is a blocking
application, while HTTPD is event-driven.
Pools in HTTPD are often bound to the lifetime of a HTTP request (event)
which means they are short-lived by design. Global data stored in pools
is of a relatively fixed size.
Whereas Subversion's client library is often used in a long-running
process context (such the Windows Explorer in case of TortoiseSVN).
The svn command line client alternates between blocking for user input
and calling into the client library. The client library will return results
to these API users and it has no idea about the lifetimes its callers require.
Which is why letting the caller manage pool lifetime is a natural thing
to do for this application.
If you are going to write some general pool usage recommendations for APR
I suppose such docs should mention how the various pool usage models relate
to the software architectures they were invented for.
Re: Using APR pools "better"
Posted by Greg Stein <gs...@gmail.com>.
On Wed, Sep 26, 2018 at 10:20 AM Daniel Shahaf <d....@daniel.shahaf.name>
wrote:
> Jim Jagielski wrote on Wed, 26 Sep 2018 11:09 -0400:
> > At ApacheCon's welcoming event last night, Greg, Sander and I were
> > chatting and Greg reminded us that the Subversion project "learned a lot
> > about using APR pools" and it seems to me that a lot of that knowledge
> > is likely missing in httpd-land. I also know that some of that has been
> > backported back to APR itself, but I am wondering (as I am wont to do),
> > if we couldn't do a better job here. I am sure that there are things in
> > svn that could be easily and readily folded back into APR w/o breaking
> > anything.
> >
> > I know that some of that influenced Greg's PoCore work, but it would be
> > really cool if someone in Subversion could maybe point out some of those
> > improvements and "lessons learned" so that both APR and httpd could
> > benefit.
>
> For starters, here's what we've already written down:
>
>
> https://subversion.apache.org/docs/community-guide/conventions.html#apr-pools
>
> I would especially point out the result_pool/scratch_pool distinction:
> each function gets *two* pools, one in which to allocate the return
> value, and one whose lifetime is defined as "the pool may be deleted at
> any time after the function returns".
>
> For example, when one calls a function declared as:
> .
> apr_status_t find_foo(foo_t *outparam, ..., apr_pool_t *result_pool,
> apr_pool_t *scratch_pool)
> .
> then on return *outparam will be allocated in result_pool, and
> apr_pool_clear(scratch_pool) may be called as soon as find_foo()
> returned.
Yup.
> Thus, scratch_pool is passed into the result_pool formal
> parameter of functions find_foo() calls, is the pool iterpools are created
> as subpools of, etc. .
>
This is a bit unclear, so skip this and go read the page :-)
> I'm sure Greg is subscribed to at least one of these lists and will add
> anything I forgot :)
All three, actually :-) ... and the pointer to the coding convention page
is exactly what I wanted to point the other groups at.
iterpool, scratch_pool, and result_pool are the KEY three concepts that we
learned while working on Subversion. Also relying on your *caller* to know
more about memory management / lifetimes. The conventions page does a
really great job of describing what we learned, and we're here to
expand/answer where it isn't (and #fixthedoc :-P ), and/or provide further
examples to clarify.
Cheers,
-g
Re: Using APR pools "better"
Posted by Greg Stein <gs...@gmail.com>.
On Wed, Sep 26, 2018 at 10:20 AM Daniel Shahaf <d....@daniel.shahaf.name>
wrote:
> Jim Jagielski wrote on Wed, 26 Sep 2018 11:09 -0400:
> > At ApacheCon's welcoming event last night, Greg, Sander and I were
> > chatting and Greg reminded us that the Subversion project "learned a lot
> > about using APR pools" and it seems to me that a lot of that knowledge
> > is likely missing in httpd-land. I also know that some of that has been
> > backported back to APR itself, but I am wondering (as I am wont to do),
> > if we couldn't do a better job here. I am sure that there are things in
> > svn that could be easily and readily folded back into APR w/o breaking
> > anything.
> >
> > I know that some of that influenced Greg's PoCore work, but it would be
> > really cool if someone in Subversion could maybe point out some of those
> > improvements and "lessons learned" so that both APR and httpd could
> > benefit.
>
> For starters, here's what we've already written down:
>
>
> https://subversion.apache.org/docs/community-guide/conventions.html#apr-pools
>
> I would especially point out the result_pool/scratch_pool distinction:
> each function gets *two* pools, one in which to allocate the return
> value, and one whose lifetime is defined as "the pool may be deleted at
> any time after the function returns".
>
> For example, when one calls a function declared as:
> .
> apr_status_t find_foo(foo_t *outparam, ..., apr_pool_t *result_pool,
> apr_pool_t *scratch_pool)
> .
> then on return *outparam will be allocated in result_pool, and
> apr_pool_clear(scratch_pool) may be called as soon as find_foo()
> returned.
Yup.
> Thus, scratch_pool is passed into the result_pool formal
> parameter of functions find_foo() calls, is the pool iterpools are created
> as subpools of, etc. .
>
This is a bit unclear, so skip this and go read the page :-)
> I'm sure Greg is subscribed to at least one of these lists and will add
> anything I forgot :)
All three, actually :-) ... and the pointer to the coding convention page
is exactly what I wanted to point the other groups at.
iterpool, scratch_pool, and result_pool are the KEY three concepts that we
learned while working on Subversion. Also relying on your *caller* to know
more about memory management / lifetimes. The conventions page does a
really great job of describing what we learned, and we're here to
expand/answer where it isn't (and #fixthedoc :-P ), and/or provide further
examples to clarify.
Cheers,
-g
Re: Using APR pools "better"
Posted by Greg Stein <gs...@gmail.com>.
On Wed, Sep 26, 2018 at 10:20 AM Daniel Shahaf <d....@daniel.shahaf.name>
wrote:
> Jim Jagielski wrote on Wed, 26 Sep 2018 11:09 -0400:
> > At ApacheCon's welcoming event last night, Greg, Sander and I were
> > chatting and Greg reminded us that the Subversion project "learned a lot
> > about using APR pools" and it seems to me that a lot of that knowledge
> > is likely missing in httpd-land. I also know that some of that has been
> > backported back to APR itself, but I am wondering (as I am wont to do),
> > if we couldn't do a better job here. I am sure that there are things in
> > svn that could be easily and readily folded back into APR w/o breaking
> > anything.
> >
> > I know that some of that influenced Greg's PoCore work, but it would be
> > really cool if someone in Subversion could maybe point out some of those
> > improvements and "lessons learned" so that both APR and httpd could
> > benefit.
>
> For starters, here's what we've already written down:
>
>
> https://subversion.apache.org/docs/community-guide/conventions.html#apr-pools
>
> I would especially point out the result_pool/scratch_pool distinction:
> each function gets *two* pools, one in which to allocate the return
> value, and one whose lifetime is defined as "the pool may be deleted at
> any time after the function returns".
>
> For example, when one calls a function declared as:
> .
> apr_status_t find_foo(foo_t *outparam, ..., apr_pool_t *result_pool,
> apr_pool_t *scratch_pool)
> .
> then on return *outparam will be allocated in result_pool, and
> apr_pool_clear(scratch_pool) may be called as soon as find_foo()
> returned.
Yup.
> Thus, scratch_pool is passed into the result_pool formal
> parameter of functions find_foo() calls, is the pool iterpools are created
> as subpools of, etc. .
>
This is a bit unclear, so skip this and go read the page :-)
> I'm sure Greg is subscribed to at least one of these lists and will add
> anything I forgot :)
All three, actually :-) ... and the pointer to the coding convention page
is exactly what I wanted to point the other groups at.
iterpool, scratch_pool, and result_pool are the KEY three concepts that we
learned while working on Subversion. Also relying on your *caller* to know
more about memory management / lifetimes. The conventions page does a
really great job of describing what we learned, and we're here to
expand/answer where it isn't (and #fixthedoc :-P ), and/or provide further
examples to clarify.
Cheers,
-g
Re: Using APR pools "better"
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Jim Jagielski wrote on Wed, 26 Sep 2018 11:09 -0400:
> At ApacheCon's welcoming event last night, Greg, Sander and I were
> chatting and Greg reminded us that the Subversion project "learned a lot
> about using APR pools" and it seems to me that a lot of that knowledge
> is likely missing in httpd-land. I also know that some of that has been
> backported back to APR itself, but I am wondering (as I am wont to do),
> if we couldn't do a better job here. I am sure that there are things in
> svn that could be easily and readily folded back into APR w/o breaking
> anything.
>
> I know that some of that influenced Greg's PoCore work, but it would be
> really cool if someone in Subversion could maybe point out some of those
> improvements and "lessons learned" so that both APR and httpd could
> benefit.
For starters, here's what we've already written down:
https://subversion.apache.org/docs/community-guide/conventions.html#apr-pools
I would especially point out the result_pool/scratch_pool distinction:
each function gets *two* pools, one in which to allocate the return
value, and one whose lifetime is defined as "the pool may be deleted at
any time after the function returns".
For example, when one calls a function declared as:
.
apr_status_t find_foo(foo_t *outparam, ..., apr_pool_t *result_pool, apr_pool_t *scratch_pool)
.
then on return *outparam will be allocated in result_pool, and
apr_pool_clear(scratch_pool) may be called as soon as find_foo()
returned. Thus, scratch_pool is passed into the result_pool formal
parameter of functions find_foo() calls, is the pool iterpools are created
as subpools of, etc. .
I'm sure Greg is subscribed to at least one of these lists and will add anything I forgot :)
Daniel
Re: Using APR pools "better"
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Jim Jagielski wrote on Wed, 26 Sep 2018 11:09 -0400:
> At ApacheCon's welcoming event last night, Greg, Sander and I were
> chatting and Greg reminded us that the Subversion project "learned a lot
> about using APR pools" and it seems to me that a lot of that knowledge
> is likely missing in httpd-land. I also know that some of that has been
> backported back to APR itself, but I am wondering (as I am wont to do),
> if we couldn't do a better job here. I am sure that there are things in
> svn that could be easily and readily folded back into APR w/o breaking
> anything.
>
> I know that some of that influenced Greg's PoCore work, but it would be
> really cool if someone in Subversion could maybe point out some of those
> improvements and "lessons learned" so that both APR and httpd could
> benefit.
For starters, here's what we've already written down:
https://subversion.apache.org/docs/community-guide/conventions.html#apr-pools
I would especially point out the result_pool/scratch_pool distinction:
each function gets *two* pools, one in which to allocate the return
value, and one whose lifetime is defined as "the pool may be deleted at
any time after the function returns".
For example, when one calls a function declared as:
.
apr_status_t find_foo(foo_t *outparam, ..., apr_pool_t *result_pool, apr_pool_t *scratch_pool)
.
then on return *outparam will be allocated in result_pool, and
apr_pool_clear(scratch_pool) may be called as soon as find_foo()
returned. Thus, scratch_pool is passed into the result_pool formal
parameter of functions find_foo() calls, is the pool iterpools are created
as subpools of, etc. .
I'm sure Greg is subscribed to at least one of these lists and will add anything I forgot :)
Daniel
Re: Using APR pools "better"
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Jim Jagielski wrote on Wed, 26 Sep 2018 11:09 -0400:
> At ApacheCon's welcoming event last night, Greg, Sander and I were
> chatting and Greg reminded us that the Subversion project "learned a lot
> about using APR pools" and it seems to me that a lot of that knowledge
> is likely missing in httpd-land. I also know that some of that has been
> backported back to APR itself, but I am wondering (as I am wont to do),
> if we couldn't do a better job here. I am sure that there are things in
> svn that could be easily and readily folded back into APR w/o breaking
> anything.
>
> I know that some of that influenced Greg's PoCore work, but it would be
> really cool if someone in Subversion could maybe point out some of those
> improvements and "lessons learned" so that both APR and httpd could
> benefit.
For starters, here's what we've already written down:
https://subversion.apache.org/docs/community-guide/conventions.html#apr-pools
I would especially point out the result_pool/scratch_pool distinction:
each function gets *two* pools, one in which to allocate the return
value, and one whose lifetime is defined as "the pool may be deleted at
any time after the function returns".
For example, when one calls a function declared as:
.
apr_status_t find_foo(foo_t *outparam, ..., apr_pool_t *result_pool, apr_pool_t *scratch_pool)
.
then on return *outparam will be allocated in result_pool, and
apr_pool_clear(scratch_pool) may be called as soon as find_foo()
returned. Thus, scratch_pool is passed into the result_pool formal
parameter of functions find_foo() calls, is the pool iterpools are created
as subpools of, etc. .
I'm sure Greg is subscribed to at least one of these lists and will add anything I forgot :)
Daniel