APR failing in low memory situations

[M Joonas Pihlaja <jp...@cc.helsinki.fi>]

Hi list,

I ran into a problem where a program I'm working on was
segfaulting when calling apr_hash_set() due to memory exhaustion.
So I was going to ask if we can have a version of apr_hash_set()
which could return a status code if it fails to add a value to a
hash.  Having a closer look at the source to APR, though, I
realised that almost none of the memory allocation calls inside
APR itself are checked for a NULL return value[1].

The thing is, there are 208 calls to apr_palloc() and 133 to
apr_pcalloc() in the source[2], so it would be quite a large job
to fix all of them.  On the up side, of all the cases I did look
at, they all seemed to be of the sort where it would be fairly
easy to add the extra checks to make them fail with APR_ENOMEM.
However, I fear the ripple effect of adding new failing states
would be much larger.  For example, all the calls to
apr_cleanup_register() would need to be changed too, since
currently that call is assumed to always succeed.

Anyway, the consistency with which memory allocations are not
checked makes me think that everyone else is using pool abort
functions in their programs.  Is this really true?  Isn't APR
supposed to be usable without them?

Best Regards,

Joonas

[1] At least among the randomish sampling of
apr_palloc()/apr_pcalloc() calls within the library I looked at.

[2] From Debian's apache2-2.2.3 package.  Corresponds to APR
version 1.2.7.

Re: APR failing in low memory situations

[Chad Fox <ch...@gigapogo.com>]

On Thu, February 22, 2007 5:11 pm, M Joonas Pihlaja wrote:
>
> On Thu, 22 Feb 2007, Bill Stoddard wrote:
>
>
>> M Joonas Pihlaja wrote:
>>
>>> However, I fear the ripple effect
>>>
>> No kidding :-)  If your app has exhausted memory, your reduced to
>> playing a never ending game of 'whack-a-mole' by attempting to tolerate
>> all of the ensuing 'bad stuff' that happens.
>>
>
> Well... it's only 'bad stuff' if you didn't see it coming. ;)
>
Sorry for the additional email, but I ran into this old thread on this topic:
http://mail-archives.apache.org/mod_mbox/apr-dev/200107.mbox/%3CPine.LNX.4.33.0107071057580.10441-100000@twinlark.arctic.org%3E

That discussion makes sense, I was just seeing apr_palloc as malloc, but
the abort function fills the gap.

Thanks
Chad

Re: APR failing in low memory situations

["William A. Rowe, Jr." <wr...@rowe-clan.net>]

Chad Fox wrote:
> 
> I'd rather play 'whack-a-mole' tearing down user X's resources and tell
> him tough luck, rather than bringing down the entire system.  The abort
> functions could work for that, but what if you just want to tell user X he
> can't have more resources and let him continue?

You don't, you want to dump X on it's ass.

which is why the abort functions could implement a thread-specific
semi-safe teardown of that threads resources.

But, you don't know that X isn't the user of 32kb, while user Y is busy
chewing away at some 1.2gb of allocation.  So that's not even particularly
effective either.

search http://mail-archives.apache.org/mod_mbox/apr-dev/ and even our
predecessor http://mail-archives.apache.org/mod_mbox/httpd-dev/ for every
argument you are about to raise and every response to such arguments.

Outside of the abort function, this is not changing.  Perhaps at some point
a recovery callback might be added to permit a particularly savvy application
to immediately release some unnecessary resources (e.g. caches) and then
retry the allocation and return on success.

But return nomem by overallocation is a crash-bug and you aren't about to
convince this experienced group of developers otherwise :)

Bill

Re: APR failing in low memory situations

[Chad Fox <ch...@gigapogo.com>]

On Thu, February 22, 2007 5:11 pm, M Joonas Pihlaja wrote:
>
> On Thu, 22 Feb 2007, Bill Stoddard wrote:
>
>> M Joonas Pihlaja wrote:
>>
>>> However, I fear the ripple effect
>>>
>> No kidding :-)  If your app has exhausted memory, your reduced to
>> playing a never ending game of 'whack-a-mole' by attempting to tolerate
>> all of the ensuing 'bad stuff' that happens.
>>
>
> Well... it's only 'bad stuff' if you didn't see it coming. ;)
>

I'm in Joonas' camp here.

If you had an application that allows users/requests to utilize seperate
sets of resources and user X got carried away and ate up a huge chunk of
memory, it's possible users A-Y are still chewing away at the resources
already allocated to them and won't be in trouble yet.

I'd rather play 'whack-a-mole' tearing down user X's resources and tell
him tough luck, rather than bringing down the entire system.  The abort
functions could work for that, but what if you just want to tell user X he
can't have more resources and let him continue?

I haven't looked at all the places where palloc is called in apr, but it
seems like in the cases of hashes, tables, arrays, etc. it would be easy
enough to add a return status as most of them have a void return.  But,
what is worse, an app that gets a segfault or one that doesn't check the
status and incorrectly assumes the apr_hash_set worked?  (i.e. "Don't
launch the missles" = true)

Chad

Re: APR failing in low memory situations

[M Joonas Pihlaja <jp...@cc.helsinki.fi>]

On Thu, 22 Feb 2007, Bill Stoddard wrote:

> M Joonas Pihlaja wrote:
> > However, I fear the ripple effect
> No kidding :-)  If your app has exhausted memory, your reduced to
> playing a never ending game of 'whack-a-mole' by attempting to tolerate
> all of the ensuing
> 'bad stuff' that happens.

Well... it's only 'bad stuff' if you didn't see it coming. ;)

> The only OS that I am personally acquainted
> with that has a robust designed-in model for dealing with low/no memory
> conditions is IBM's zOS.

It's not like a low memory situation for one program means that
the entire operating system has to croak any time soon, not even
under UNIX.  I do agree, though, that every layer of an
application must have a robust strategy for dealing with resource
limits, or the whole program will come down like a house of cards
at an inappropriate moment.

Cheers,

Joonas

Re: APR failing in low memory situations

[Bill Stoddard <bi...@wstoddard.com>]

M Joonas Pihlaja wrote:
> Hi list,
>
> I ran into a problem where a program I'm working on was
> segfaulting when calling apr_hash_set() due to memory exhaustion.
> So I was going to ask if we can have a version of apr_hash_set()
> which could return a status code if it fails to add a value to a
> hash.  Having a closer look at the source to APR, though, I
> realised that almost none of the memory allocation calls inside
> APR itself are checked for a NULL return value[1].
>
> The thing is, there are 208 calls to apr_palloc() and 133 to
> apr_pcalloc() in the source[2], so it would be quite a large job
> to fix all of them.  On the up side, of all the cases I did look
> at, they all seemed to be of the sort where it would be fairly
> easy to add the extra checks to make them fail with APR_ENOMEM.
>   

> However, I fear the ripple effect 
No kidding :-)  If your app has exhausted memory, your reduced to 
playing a never ending game of 'whack-a-mole' by attempting to tolerate 
all of the ensuing
'bad stuff' that happens.   The only OS that I am personally acquainted 
with that has a robust designed-in model for dealing with low/no memory 
conditions is IBM's zOS.  I don't see us ever adding null pointer checks 
to apr_palloc/apr_pcalloc returns.

Bill

Bill