You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apr.apache.org by Justin Erenkrantz <je...@ebuilt.com> on 2001/07/07 18:14:56 UTC
Re: cvs commit: apr/memory/unix apr_sms.c apr_sms_trivial.c sms_private.h
On Sat, Jul 07, 2001 at 12:26:32PM -0000, dreid@apache.org wrote:
> dreid 01/07/07 05:26:32
>
> Modified: include apr_sms.h
> memory/unix apr_sms.c apr_sms_trivial.c sms_private.h
> Log:
> Add the abort function into sms. This is largely to keep compatability
> with pools and at present we don't do anything more than get/set, but
> that's easily fixed.
>
> Question is, do we still want this capability?
I don't think there is any benefit to this. But, someone else might
be able to chime in and say why we need this functionality. -- justin
Re: cvs commit: apr/memory/unix apr_sms.c apr_sms_trivial.c
Posted by dr...@jetnet.co.uk.
> now... you guys may have palloc() returning NULL because you've put a
> layer of indirection into pools so that you can have pools on top of
> exotic storages. if that's the case, then i really hope that one of
> those exotic pools is never passed to an httpd routine expecting a
> plain old memory pool. 'cause you'll waste a lot of cpu cycles putting
> NULL checks everywhere.
agreed
> i didn't look to see what happenned where the old abort() was.
> anything which manages to squeeze out a log message (or exit(1) trying)
> is totally cool.
which is what we'll add. i didn't see a default abort in the pools code,
but i may not have looked hard enough, or i'd have added one already. i
guess it'll just dump a message to stderr and then exit, just like it does
already in some routines.
david
Re: cvs commit: apr/memory/unix apr_sms.c apr_sms_trivial.c
sms_private.h
Posted by dean gaudet <de...@arctic.org>.
On Sat, 7 Jul 2001, Aaron Bannert wrote:
> > > I don't think there is any benefit to this. But, someone else might
> > > be able to chime in and say why we need this functionality. -- justin
> >
> > how do you handle out of memory?
> >
> > palloc is not permitted to return NULL for performance reasons. (well
> > that plus the massive amounts of legacy code)
> >
> > of course this whole topic is a mess and really wants proper exception
> > handling... which we don't have in a convenient form in C :)
>
> palloc() will never return NULL? Now I'm curious how out-of-memory is
> handled. I guess I (wrongly) assumed palloc() would return NULL the
> way malloc() does. *shrug*
palloc is not malloc.
just grep in httpd-2.0 for palloc and notice how nothing checks its return
value. (ok maybe something does, i didn't look close enough. look at
apache-1.3 if you don't believe me :).
the truth is that it's pointless to check for out-of-memory return values
in a portable program anyhow. the various sane operating systems out
there use optimistic memory allocation (i.e. linux) because pessimistic
memory allocation (i.e. solaris) sucks ass whenever you work with a
fork()d server. solaris boxes need massive swap partitions to handle
bloated mod_perl applications which consist of a few dozen fork()s of the
same process -- because solaris wants to commit backing store for every
single page which is marked copy-on-write.
so if you've got, say, 20MB of memory which is shared and never modified
between 20 processes, then solaris wants 400MB of backing store. linux
won't allocate backing store until a write actually occurs on that page.
this means linux won't give you an "out of memory error" until you try to
write to a page and it can't allocate a page of storage. how does it
signal the error? it sends you a SIGSEGV.
i forget where the various BSDs live in the spectrum of optimistic vs.
pessimistic memory allocation.
i also shouldn't rag TOO hard on solaris, there is a way to do optimistic
memory allocation (using mmap()), but that's not what malloc() uses.
it's also in good company, there's other pessimistic beasties out there.
so anyhow, malloc() can give you a non-NULL pointer for storage that just
plain doesn't exist, and you can waste your time testing it's non-NULL
only to get a SIGSEGV when you try to actually use it.
it's better to just abort when you get a NULL pointer :) you're really in
trouble when that happens. in order to unwind from a NULL on memory
allocation you would have to have carefully allocated (and written to!)
all the memory (and stack space) required by your recovery mechanism.
nobody codes that well. you might try to write your own code that well,
but what about all the libraries you use?
now... you guys may have palloc() returning NULL because you've put a
layer of indirection into pools so that you can have pools on top of
exotic storages. if that's the case, then i really hope that one of those
exotic pools is never passed to an httpd routine expecting a plain old
memory pool. 'cause you'll waste a lot of cpu cycles putting NULL checks
everywhere.
i didn't look to see what happenned where the old abort() was. anything
which manages to squeeze out a log message (or exit(1) trying) is totally
cool.
-dean
Re: cvs commit: apr/memory/unix apr_sms.c apr_sms_trivial.c sms_private.h
Posted by Aaron Bannert <aa...@ebuilt.com>.
> > I don't think there is any benefit to this. But, someone else might
> > be able to chime in and say why we need this functionality. -- justin
>
> how do you handle out of memory?
>
> palloc is not permitted to return NULL for performance reasons. (well
> that plus the massive amounts of legacy code)
>
> of course this whole topic is a mess and really wants proper exception
> handling... which we don't have in a convenient form in C :)
palloc() will never return NULL? Now I'm curious how out-of-memory is
handled. I guess I (wrongly) assumed palloc() would return NULL the
way malloc() does. *shrug*
-aaron
Re: cvs commit: apr/memory/unix apr_sms.c apr_sms_trivial.c
sms_private.h
Posted by dean gaudet <de...@arctic.org>.
On Sat, 7 Jul 2001, Justin Erenkrantz wrote:
> On Sat, Jul 07, 2001 at 12:26:32PM -0000, dreid@apache.org wrote:
> > dreid 01/07/07 05:26:32
> >
> > Modified: include apr_sms.h
> > memory/unix apr_sms.c apr_sms_trivial.c sms_private.h
> > Log:
> > Add the abort function into sms. This is largely to keep compatability
> > with pools and at present we don't do anything more than get/set, but
> > that's easily fixed.
> >
> > Question is, do we still want this capability?
>
> I don't think there is any benefit to this. But, someone else might
> be able to chime in and say why we need this functionality. -- justin
how do you handle out of memory?
palloc is not permitted to return NULL for performance reasons. (well
that plus the massive amounts of legacy code)
of course this whole topic is a mess and really wants proper exception
handling... which we don't have in a convenient form in C :)
-dean
Re: cvs commit: apr/memory/unix apr_sms.c apr_sms_trivial.c
sms_private.h
Posted by rb...@covalent.net.
On Sun, 8 Jul 2001, Aaron Bannert wrote:
> On Sat, Jul 07, 2001 at 05:14:47PM -0700, Justin Erenkrantz wrote:
> [...]
> > Error-checking
> > is something that we pointedly avoid doing in APR.
> [...]
>
> Please tell me this is not a general policy, and that APR has a small
> amount of error checking. Pretty please? I don't advocate pedantic
> error checking, but it should have at least enough to make it easier for
> developers to find their bugs while coding. (isn't this the difference
> between "debug" and "release" versions? Efficiency at the expense of
> minor usability improvements??)
>
> (yes, I'm revisiting a thread from a few weeks ago)
There is minimal error checking in APR. Obviously we return error codes,
so we do know when something has gone wrong. We do not check to ensure
that a user has passed in valid data, because that is a waste of time.
Better to let the program die, and allow the programmer to deal with the
seg fault.
Ryan
_____________________________________________________________________________
Ryan Bloom rbb@apache.org
Covalent Technologies rbb@covalent.net
-----------------------------------------------------------------------------
Re: cvs commit: apr/memory/unix apr_sms.c apr_sms_trivial.c sms_private.h
Posted by Aaron Bannert <aa...@ebuilt.com>.
On Sat, Jul 07, 2001 at 05:14:47PM -0700, Justin Erenkrantz wrote:
[...]
> Error-checking
> is something that we pointedly avoid doing in APR.
[...]
Please tell me this is not a general policy, and that APR has a small
amount of error checking. Pretty please? I don't advocate pedantic
error checking, but it should have at least enough to make it easier for
developers to find their bugs while coding. (isn't this the difference
between "debug" and "release" versions? Efficiency at the expense of
minor usability improvements??)
(yes, I'm revisiting a thread from a few weeks ago)
-aaron
Re: cvs commit: apr/memory/unix apr_sms.c apr_sms_trivial.c
sms_private.h
Posted by rb...@covalent.net.
On Sat, 7 Jul 2001, Justin Erenkrantz wrote:
> On Sat, Jul 07, 2001 at 04:01:35PM -0700, dean gaudet wrote:
> > what's it really matter? trying to clean up from OOM is near impossible.
It doesn't matter if it's impossible or not. The problem is that we are
in a library, so we can't just write to stderr by default. You have no
way of knowing what stderr is, or that it even exists.
> Which brings me back to my original question: "Why do we have an abort
> function?" As you said (I was originally thinking this, but admittedly,
> I didn't bring this up), we can't do much. About the only thing we can
> do is exit. We *might* be able to make a log entry somewhere, but that
> depends if we need memory to do the printf or whatnot.
The point is that the original code from Apache 1.3 wrote a message to
stderr, and aborted. The new code, has that option, but the default is to
return an error condition to the caller. Again, you can't just write to
stderr, because you don't know that stderr even exists anymore.
> And, if palloc returns junk - NULL or non-NULL, we're bound to segfault
> at some point - which takes care of our process anyway. Error-checking
> is something that we pointedly avoid doing in APR.
>
> I'm not still sure what the abort function gets us. -- justin
The simple ability to let the program decide how to handle this error
condition. This was pain-stakingly decided over a year ago, and before we
decide to take it out, please go back and read the mailing list archives
for November 1999.
Apache wants to write an error message and abort. I will -1 having a
general purpose library do that by default, because it is VERY
inconsiderate to the calling program. A general purpose library must
allow the program to know what is going on before it just dies.
I don't care if 99% of the time it is impossible to recover. This isn't
about recovery. It is about not allowing the library to just kill the
program.
Ryan
_____________________________________________________________________________
Ryan Bloom rbb@apache.org
Covalent Technologies rbb@covalent.net
-----------------------------------------------------------------------------
Re: cvs commit: apr/memory/unix apr_sms.c apr_sms_trivial.c sms_private.h
Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Sat, Jul 07, 2001 at 04:01:35PM -0700, dean gaudet wrote:
> what's it really matter? trying to clean up from OOM is near impossible.
Which brings me back to my original question: "Why do we have an abort
function?" As you said (I was originally thinking this, but admittedly,
I didn't bring this up), we can't do much. About the only thing we can
do is exit. We *might* be able to make a log entry somewhere, but that
depends if we need memory to do the printf or whatnot.
And, if palloc returns junk - NULL or non-NULL, we're bound to segfault
at some point - which takes care of our process anyway. Error-checking
is something that we pointedly avoid doing in APR.
I'm not still sure what the abort function gets us. -- justin
Re: cvs commit: apr/memory/unix apr_sms.c apr_sms_trivial.c
sms_private.h
Posted by dean gaudet <de...@arctic.org>.
On Sat, 7 Jul 2001 rbb@covalent.net wrote:
> On Sat, 7 Jul 2001, Justin Erenkrantz wrote:
>
> > On Sat, Jul 07, 2001 at 12:26:32PM -0000, dreid@apache.org wrote:
> > > dreid 01/07/07 05:26:32
> > >
> > > Modified: include apr_sms.h
> > > memory/unix apr_sms.c apr_sms_trivial.c sms_private.h
> > > Log:
> > > Add the abort function into sms. This is largely to keep compatability
> > > with pools and at present we don't do anything more than get/set, but
> > > that's easily fixed.
> > >
> > > Question is, do we still want this capability?
> >
> > I don't think there is any benefit to this. But, someone else might
> > be able to chime in and say why we need this functionality. -- justin
>
> This MUST stay. The reason is simple. A well behaved library will not
> exit, or print a message to stderr. However, because APR needs to
> maintain backwards compat with Apache 1.3, when we can't allocate memory,
> we print to stderr, and abort.
>
> Unless there is another way to use APR and provide both options, the abort
> function must remain.
what's it really matter? trying to clean up from OOM is near impossible.
the task will be shot on linux or anywhere else with optimistic memory
allocation in many situations.
linux 2.4 changes the OOM situation slightly, but only in the details as
to which task gets shot. (not necessarily the one which tried to access
the last available free page.)
and like i said in my other message, you have no idea from NULL vs.
non-NULL as to whether you're out of memory yet. in fact, malloc may
brk() to get more memory, then segfault while trying to lay down its
structure into that memory. (linux malloc of >=128kb blocks use mmap
directly and they're page aligned with no malloc structures, so the
segfault doesn't happen until the application touches it... for the "why"
of this study the mremap() syscall and try remallocing big regions on
linux :)
oh also, you might fault trying to grow the stack and that could take the
task out.
also, a box which is OOM desperately needs memory in order to function.
either your task is going to die, or something else is. the box is
probably so deep into swap that it's unuseable, and the faster you die the
better for everyone.
one might ask "but what about all those resources i need to unwind and
release" ... and to that i'll just say: power failure. if resource mgmt
is so important then you've already considered power failures and you've
concluded that you need to use transaction logging techniques. (this is
also why i really like things which are freed up and released properly on
their own when a process exits... like kernel-based locks, and not like
pthread mutexes :)
i kind of gave up on memory allocation checking. i'd consider it if i was
coding in ada or some language with appropriate exception mechanisms.
but not in C.
but i could just be jaded :)
-dean
Re: cvs commit: apr/memory/unix apr_sms.c apr_sms_trivial.c
sms_private.h
Posted by rb...@covalent.net.
On Sat, 7 Jul 2001, Justin Erenkrantz wrote:
> On Sat, Jul 07, 2001 at 12:26:32PM -0000, dreid@apache.org wrote:
> > dreid 01/07/07 05:26:32
> >
> > Modified: include apr_sms.h
> > memory/unix apr_sms.c apr_sms_trivial.c sms_private.h
> > Log:
> > Add the abort function into sms. This is largely to keep compatability
> > with pools and at present we don't do anything more than get/set, but
> > that's easily fixed.
> >
> > Question is, do we still want this capability?
>
> I don't think there is any benefit to this. But, someone else might
> be able to chime in and say why we need this functionality. -- justin
This MUST stay. The reason is simple. A well behaved library will not
exit, or print a message to stderr. However, because APR needs to
maintain backwards compat with Apache 1.3, when we can't allocate memory,
we print to stderr, and abort.
Unless there is another way to use APR and provide both options, the abort
function must remain.
Ryan
_____________________________________________________________________________
Ryan Bloom rbb@apache.org
Covalent Technologies rbb@covalent.net
-----------------------------------------------------------------------------