reliable piped logs

[Dean Gaudet <dg...@arctic.org>]

So the reason I've hated to recommend using | to a logger before is
that it's totally unreliable.  If for some reason the child should die,
bad things happen.  Ideally the parent should be able to respawn the
logging child and no hits should be lost.

The children cannot do the spawning (there's the problem of
synchronization, and permissions problems), and they need some way to get
a new file handle for the new child.  Here's a few ways with the
pros and cons I can think of:

- use descriptor passing to pass handle to each child

    Given that NCSA uses descriptor passing, this works well on most
    architectures.  It's slow, but it's also not something that's done
    frequently.  It would require the parent to have a fd open to
    each child, damn near impossible if you ask me

- use a named pipe

    Child can reopen the named pipe if the logger dies.

    I'm not sure that you can get the "multiple-writers, one-reader"
    semantic on named pipes.  It seems to work in some cases (i.e. all
    writers blocked trying to open it before the reader opens it)...
    but fails in other cases.  So we'd probably have to hack it to have
    a named pipe for each child slot, and the logging child would need
    to open up each named pipe... and would have to always try to re-open
    them ... and wouldn't be able to do that without blocking, and would
    totally suck.

- use unix domain sockets

    Child can reopen the socket if the logger dies.

    Similar problems to the last one when using SOCK_STREAM sockets --
    the logger needs to do listen()/accept() crap on the unix socket
    and have a fd for each child.  But SOCK_DGRAM offers some hope.
    The logger would just read() datagrams, not caring who sent them.

    Unix domain sockets (and probably named pipes) have some interesting
    problems on various unixes -- regardless of the file owner/group and
    permission bits any process can open the socket/pipe.  The only way
    to protect them is to nest them in a directory with the appropriate
    permissions.  This is a very common problem (present in BSD up through
    most of the 4.3s I think).  Regardless, the uid the children runs
    as need to be able to write the socket ...  which is bad because
    then CGIs could potentially write the socket.

    SOCK_DGRAM isn't reliable, but in practice only loses packets (in
    unix domain) if the socket runs out of buffer space ... i.e. the log
    child takes too long to read.

- un-named pipes and graceful restart

    Proceed as we do now and if the logger dies then perform a
    graceful restart.  Will lose the log of all the hits that children
    are attempting to log when the graceful hits.

    This could be helped by having an "overflow" file, much like how INN
    handles its feeds.  When a child discovers that the logger is dead,
    or that it would have to block while writing to the logger, then write
    the data to another file handle into an overflow file.  The parent
    opens both the logger pipe and the overflow file at init time.

    There's a bit of work to properly get the logger to handle the overflow
    file ... it's not easy (or desirable imho) to get db-like reliability
    here, but it's possible to get a bit better than nothing.

I'm partial to the last one.

Thoughts?  Additions?

Dean

Re: reliable piped logs

[Dean Gaudet <dg...@arctic.org>]

On Sun, 6 Jul 1997, Marc Slemko wrote:
> > Here's another one... have the parent open a pipe() to the logger, and
> > a separate pipe() that multiplexes the children.  The parent shuffles
> > bytes from the child pipe into the logger pipe.  It's ok to puke now,
> > I'm just being thorough.
> 
> You know, that one isn't actually so bad if you ignore the fact that you
> are adding another copy.  Unfortunately, that fact is a bit hard to
> ignore.

Hey maybe this is going somewhere.  Suppose the parent calls
pipe(logger_fds).  When it forks the logger child it close(logger_fds[1]).
When it forks the httpd children it close(logger_fds[0]).  If the logger
dies, the parent still has an open handle to the pipe in write mode, so
the children won't get EPIPE.  The parent can just fork another logger.
The parent forks another logger when it sees the logger pid come back
from wait().

I'm going to go write a test program for this.

Dean

Re: reliable piped logs

[Marc Slemko <ma...@worldgate.com>]

On Sun, 6 Jul 1997, Dean Gaudet wrote:

> On Sun, 6 Jul 1997, Marc Slemko wrote:
> 
> > Note that I also want to be able to do something similar for the listen
> > sockets.  On some OSes (particularly Linux and Solaris) we can get in
> > endless loops of accept() failures when something bad happens to the
> > socket, so it would be nice to detect that and close/reopen it.
> 
> The endless loop of select() thing seems to have disappeared with 1.2
> though, hasn't it?  I remember seeing reports against early betas but Roy
> rewrote child_main and it seemed to cure it.

Erm... accept(), not select().

No, I think I still have reports under 1.2.  Just remembered, Unixware was
the other OS I saw a lot of reports about it on.  The basic problem with
the current code is that if the socket gets messed up in any way, Apache
will never figure it out and reopen it.  Unfortunately, this isn't easy to
fix which is why it isn't. 

We can see how it goes with 1.2... 

> 
> > > Thoughts?  Additions?
> > 
> > I guess I prefer the last one, but I'm not overly happy with any of them.
> > I'm not sure I trust using a child process in any case, no matter how
> > reliable it is.  That could just be my paranoia though.
> 
> Here's another one... have the parent open a pipe() to the logger, and
> a separate pipe() that multiplexes the children.  The parent shuffles
> bytes from the child pipe into the logger pipe.  It's ok to puke now,
> I'm just being thorough.

You know, that one isn't actually so bad if you ignore the fact that you
are adding another copy.  Unfortunately, that fact is a bit hard to
ignore.

> Oh yeah, on the unix-domain one, I might be wrong -- the children may
> not ever need to "reopen" it if you can connect() and then pass the
> connected socket to the children.  A connected dgram socket means
> you don't have to specify the receiver when you're writing to it.
> This may run into portability problems -- hopefully it's implemented
> with an access check at connect() time rather than at write() time.
> This could make the unix-domain one a better choice, depending on how
> you feel about the possibility of losing a few datagrams in high
> loads.

Hmm.  Interesting idea.  Still don't buy the reliability though.  OTOH, I
don't buy into syslog either but I (and nearly everyone else in the world) 
use it all the time. 

Re: reliable piped logs

[Dean Gaudet <dg...@arctic.org>]

On Sun, 6 Jul 1997, Marc Slemko wrote:

> Note that I also want to be able to do something similar for the listen
> sockets.  On some OSes (particularly Linux and Solaris) we can get in
> endless loops of accept() failures when something bad happens to the
> socket, so it would be nice to detect that and close/reopen it.

The endless loop of select() thing seems to have disappeared with 1.2
though, hasn't it?  I remember seeing reports against early betas but Roy
rewrote child_main and it seemed to cure it.

> > Thoughts?  Additions?
> 
> I guess I prefer the last one, but I'm not overly happy with any of them.
> I'm not sure I trust using a child process in any case, no matter how
> reliable it is.  That could just be my paranoia though.

Here's another one... have the parent open a pipe() to the logger, and
a separate pipe() that multiplexes the children.  The parent shuffles
bytes from the child pipe into the logger pipe.  It's ok to puke now,
I'm just being thorough.

Oh yeah, on the unix-domain one, I might be wrong -- the children may
not ever need to "reopen" it if you can connect() and then pass the
connected socket to the children.  A connected dgram socket means
you don't have to specify the receiver when you're writing to it.
This may run into portability problems -- hopefully it's implemented
with an access check at connect() time rather than at write() time.
This could make the unix-domain one a better choice, depending on how
you feel about the possibility of losing a few datagrams in high
loads.

> Oh yes, all the child process stuff should be abstracted out anyway.
> Isn't there a seperate routine for each logging module that wants to use
> it?

Definitely.

Dean

Re: reliable piped logs

[Marc Slemko <ma...@worldgate.com>]

On Sun, 6 Jul 1997, Dean Gaudet wrote:

> So the reason I've hated to recommend using | to a logger before is
> that it's totally unreliable.  If for some reason the child should die,
> bad things happen.  Ideally the parent should be able to respawn the
> logging child and no hits should be lost.

Note that I also want to be able to do something similar for the listen
sockets.  On some OSes (particularly Linux and Solaris) we can get in
endless loops of accept() failures when something bad happens to the
socket, so it would be nice to detect that and close/reopen it.

> 
> The children cannot do the spawning (there's the problem of
> synchronization, and permissions problems), and they need some way to get
> a new file handle for the new child.  Here's a few ways with the
> pros and cons I can think of:
> 

> - un-named pipes and graceful restart
> 
>     Proceed as we do now and if the logger dies then perform a
>     graceful restart.  Will lose the log of all the hits that children
>     are attempting to log when the graceful hits.
> 
>     This could be helped by having an "overflow" file, much like how INN
>     handles its feeds.  When a child discovers that the logger is dead,
>     or that it would have to block while writing to the logger, then write
>     the data to another file handle into an overflow file.  The parent
>     opens both the logger pipe and the overflow file at init time.
> 
>     There's a bit of work to properly get the logger to handle the overflow
>     file ... it's not easy (or desirable imho) to get db-like reliability
>     here, but it's possible to get a bit better than nothing.
> 
> I'm partial to the last one.
> 
> Thoughts?  Additions?

I guess I prefer the last one, but I'm not overly happy with any of them.
I'm not sure I trust using a child process in any case, no matter how
reliable it is.  That could just be my paranoia though.

I still think adding a bit to mod_log_config would be good.  

Oh yes, all the child process stuff should be abstracted out anyway.
Isn't there a seperate routine for each logging module that wants to use
it?