You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Chris Darroch <ch...@pearsoncmg.com> on 2012/12/18 23:22:16 UTC
rotatelogs input from named pipe
Hi --
I recently needed to wire up httpd's rotatelogs to a named pipe (a FIFO)
instead of to stdin. For context, we have a non-httpd server process
configured to write to the pipe instead of a log file; further, this
server likes to intermittently open, write to, and close its various
log files.
I wanted/needed to open the pipe in non-blocking mode, so rotatelogs
would start right away (we're using rotatelogs -cf) instead of blocking
until a writer process happened to open the pipe to write some log data.
Also, because the server process repeatedly opens and closes its
log files/pipes, a normal reader will exit on EOF after the first close,
and therefore miss all subsequent writes.
I played around for some time with variants on "tail -f pipe | rotatelogs"
and "cat pipe | rotatelogs" but both have problems. The tail -f seems
to miss data, especially at first; I can't explain that but it's what
I observed. The cat command works to capture all data while running,
but it blocks at startup and (worse for us) exits on the first EOF.
What I really wanted was a simpler command line, passing the pipe to
rotatelogs directly, and having it (a) open in non-blocking mode so
the -cf options are effective right away, and (b) ignore EOFs until I kill
it explicitly. The latter is ungainly but that's how the server also
has to be stopped, via kill -TERM, so it's acceptable in this case for us.
First I puzzled over how to open a pipe in non-blocking mode from the
start using APR, and ended up with a set of patches for that; see
http://marc.info/?l=apr-dev&m=135586687408393&w=2 for the details,
but the gist is an apr_file_namedpipe_open() function to do this.
Probably not sufficiently full of trademark portable goodness, but
effective for the moment.
Next the following patch adds a "-i inputpipe [-o]" set of options,
when possible, so that rotatelogs will listen to the pipe instead of
stdin and will ignore EOFs.
I'm posting both the APR and httpd patches because I thought it was
an interesting issue, and I suspect there are alternate and perhaps
better approaches; certainly the APR patch isn't ideal. But after
a long time spent digging into the source code for tail and cat and
trying many variants, this now seems to work smoothly for us, at least.
Chris.
===================================================================
--- support/rotatelogs.c.orig 2012-12-18 13:44:02.000000000 -0800
+++ support/rotatelogs.c 2012-12-18 14:11:56.000000000 -0800
@@ -67,6 +67,8 @@
int truncate;
const char *linkfile;
const char *postrotate_prog;
+ const char *inputpipe;
+ int ignore_eof;
#if APR_FILES_AS_SOCKETS
int create_empty;
#endif
@@ -101,9 +103,9 @@
}
fprintf(stderr,
#if APR_FILES_AS_SOCKETS
- "Usage: %s [-v] [-l] [-L linkname] [-p prog] [-f] [-t] [-e] [-c] <logfile> "
+ "Usage: %s [-v] [-l] [-L linkname] [-p prog] [-i inputpipe [-o]] [-f] [-t] [-e] [-c] <logfile> "
#else
- "Usage: %s [-v] [-l] [-L linkname] [-p prog] [-f] [-t] [-e] <logfile> "
+ "Usage: %s [-v] [-l] [-L linkname] [-p prog] [-i inputpipe] [-f] [-t] [-e] <logfile> "
#endif
"{<rotation time in seconds>|<rotation size>(B|K|M|G)} "
"[offset minutes from UTC]\n\n",
@@ -137,7 +139,9 @@
" -f Force opening of log on program start.\n"
" -t Truncate logfile instead of rotating, tail friendly.\n"
" -e Echo log to stdout for further processing.\n"
+ " -i path Use specified named pipe as input.\n"
#if APR_FILES_AS_SOCKETS
+ " -o Hold input pipe open when writers exit (with -i only).\n"
" -c Create log even if it is empty.\n"
#endif
"\n"
@@ -194,7 +198,9 @@
fprintf(stderr, "Rotation verbose: %12s\n", config->verbose ? "yes" : "no");
#if APR_FILES_AS_SOCKETS
fprintf(stderr, "Rotation create empty logs: %12s\n", config->create_empty ? "yes" : "no");
+ fprintf(stderr, "Input pipe held open: %12s\n", config->ignore_eof ? "yes" : "no");
#endif
+ fprintf(stderr, "Input pipe: %21s\n", config->inputpipe);
fprintf(stderr, "Rotation file name: %21s\n", config->szLogRoot);
fprintf(stderr, "Post-rotation prog: %21s\n", config->postrotate_prog);
}
@@ -529,9 +535,9 @@
apr_pool_create(&status.pool, NULL);
apr_getopt_init(&opt, status.pool, argc, argv);
#if APR_FILES_AS_SOCKETS
- while ((rv = apr_getopt(opt, "lL:p:ftvec", &c, &opt_arg)) == APR_SUCCESS) {
+ while ((rv = apr_getopt(opt, "lL:p:i:oftvec", &c, &opt_arg)) == APR_SUCCESS) {
#else
- while ((rv = apr_getopt(opt, "lL:p:ftve", &c, &opt_arg)) == APR_SUCCESS) {
+ while ((rv = apr_getopt(opt, "lL:p:i:ftve", &c, &opt_arg)) == APR_SUCCESS) {
#endif
switch (c) {
case 'l':
@@ -543,6 +549,12 @@
case 'p':
config.postrotate_prog = opt_arg;
break;
+ case 'i':
+ config.inputpipe = opt_arg;
+ break;
+ case 'o':
+ config.ignore_eof = 1;
+ break;
case 'f':
config.force_open = 1;
break;
@@ -588,9 +600,25 @@
config.use_strftime = (strchr(config.szLogRoot, '%') != NULL);
- if (apr_file_open_stdin(&f_stdin, status.pool) != APR_SUCCESS) {
- fprintf(stderr, "Unable to open stdin\n");
- exit(1);
+ if (config.inputpipe) {
+ rv = apr_file_namedpipe_open(&f_stdin, config.inputpipe,
+ APR_FOPEN_READ |
+ (config.ignore_eof ? APR_FOPEN_WRITE : 0),
+ APR_FULL_NONBLOCK, status.pool);
+
+ if (rv != APR_SUCCESS) {
+ fprintf(stderr, "Unable to open pipe %s\n", config.inputpipe);
+ exit(1);
+ }
+ } else {
+ if (config.ignore_eof) {
+ usage(argv[0], "-o requires -i <inputpipe>");
+ }
+
+ if (apr_file_open_stdin(&f_stdin, status.pool) != APR_SUCCESS) {
+ fprintf(stderr, "Unable to open stdin\n");
+ exit(1);
+ }
}
if (apr_file_open_stdout(&f_stdout, status.pool) != APR_SUCCESS) {
@@ -637,7 +665,11 @@
if (pollret == APR_SUCCESS) {
rv = apr_file_read(f_stdin, buf, &nRead);
if (APR_STATUS_IS_EOF(rv)) {
- break;
+ if (config.ignore_eof) {
+ continue;
+ } else {
+ break;
+ }
}
else if (rv != APR_SUCCESS) {
exit(3);
===================================================================
--
GPG Key ID: 088335A9
GPG Key Fingerprint: 86CD 3297 7493 75BC F820 6715 F54F E648 0883 35A9
Re: rotatelogs input from named pipe
Posted by Igor Galić <i....@brainsware.org>.
Hey folks,
has anyone followed up on Chris' patch?
----- Original Message -----
> Hi --
>
> I recently needed to wire up httpd's rotatelogs to a named pipe (a
> FIFO)
> instead of to stdin. For context, we have a non-httpd server process
> configured to write to the pipe instead of a log file; further, this
> server likes to intermittently open, write to, and close its various
> log files.
>
> I wanted/needed to open the pipe in non-blocking mode, so
> rotatelogs
> would start right away (we're using rotatelogs -cf) instead of
> blocking
> until a writer process happened to open the pipe to write some log
> data.
>
> Also, because the server process repeatedly opens and closes its
> log files/pipes, a normal reader will exit on EOF after the first
> close,
> and therefore miss all subsequent writes.
>
> I played around for some time with variants on "tail -f pipe |
> rotatelogs"
> and "cat pipe | rotatelogs" but both have problems. The tail -f
> seems
> to miss data, especially at first; I can't explain that but it's what
> I observed. The cat command works to capture all data while running,
> but it blocks at startup and (worse for us) exits on the first EOF.
>
> What I really wanted was a simpler command line, passing the pipe
> to
> rotatelogs directly, and having it (a) open in non-blocking mode so
> the -cf options are effective right away, and (b) ignore EOFs until I
> kill
> it explicitly. The latter is ungainly but that's how the server also
> has to be stopped, via kill -TERM, so it's acceptable in this case
> for us.
>
> First I puzzled over how to open a pipe in non-blocking mode from
> the
> start using APR, and ended up with a set of patches for that; see
> http://marc.info/?l=apr-dev&m=135586687408393&w=2 for the details,
> but the gist is an apr_file_namedpipe_open() function to do this.
> Probably not sufficiently full of trademark portable goodness, but
> effective for the moment.
>
> Next the following patch adds a "-i inputpipe [-o]" set of
> options,
> when possible, so that rotatelogs will listen to the pipe instead of
> stdin and will ignore EOFs.
>
> I'm posting both the APR and httpd patches because I thought it
> was
> an interesting issue, and I suspect there are alternate and perhaps
> better approaches; certainly the APR patch isn't ideal. But after
> a long time spent digging into the source code for tail and cat and
> trying many variants, this now seems to work smoothly for us, at
> least.
>
> Chris.
>
> ===================================================================
> --- support/rotatelogs.c.orig 2012-12-18 13:44:02.000000000 -0800
> +++ support/rotatelogs.c 2012-12-18 14:11:56.000000000 -0800
> @@ -67,6 +67,8 @@
> int truncate;
> const char *linkfile;
> const char *postrotate_prog;
> + const char *inputpipe;
> + int ignore_eof;
> #if APR_FILES_AS_SOCKETS
> int create_empty;
> #endif
> @@ -101,9 +103,9 @@
> }
> fprintf(stderr,
> #if APR_FILES_AS_SOCKETS
> - "Usage: %s [-v] [-l] [-L linkname] [-p prog] [-f] [-t]
> [-e] [-c] <logfile> "
> + "Usage: %s [-v] [-l] [-L linkname] [-p prog] [-i
> inputpipe [-o]] [-f] [-t] [-e] [-c] <logfile> "
> #else
> - "Usage: %s [-v] [-l] [-L linkname] [-p prog] [-f] [-t]
> [-e] <logfile> "
> + "Usage: %s [-v] [-l] [-L linkname] [-p prog] [-i
> inputpipe] [-f] [-t] [-e] <logfile> "
> #endif
> "{<rotation time in seconds>|<rotation size>(B|K|M|G)} "
> "[offset minutes from UTC]\n\n",
> @@ -137,7 +139,9 @@
> " -f Force opening of log on program start.\n"
> " -t Truncate logfile instead of rotating, tail
> friendly.\n"
> " -e Echo log to stdout for further
> processing.\n"
> + " -i path Use specified named pipe as input.\n"
> #if APR_FILES_AS_SOCKETS
> + " -o Hold input pipe open when writers exit (with
> -i only).\n"
> " -c Create log even if it is empty.\n"
> #endif
> "\n"
> @@ -194,7 +198,9 @@
> fprintf(stderr, "Rotation verbose: %12s\n",
> config->verbose ? "yes" : "no");
> #if APR_FILES_AS_SOCKETS
> fprintf(stderr, "Rotation create empty logs: %12s\n",
> config->create_empty ? "yes" : "no");
> + fprintf(stderr, "Input pipe held open: %12s\n",
> config->ignore_eof ? "yes" : "no");
> #endif
> + fprintf(stderr, "Input pipe: %21s\n",
> config->inputpipe);
> fprintf(stderr, "Rotation file name: %21s\n",
> config->szLogRoot);
> fprintf(stderr, "Post-rotation prog: %21s\n",
> config->postrotate_prog);
> }
> @@ -529,9 +535,9 @@
> apr_pool_create(&status.pool, NULL);
> apr_getopt_init(&opt, status.pool, argc, argv);
> #if APR_FILES_AS_SOCKETS
> - while ((rv = apr_getopt(opt, "lL:p:ftvec", &c, &opt_arg)) ==
> APR_SUCCESS) {
> + while ((rv = apr_getopt(opt, "lL:p:i:oftvec", &c, &opt_arg)) ==
> APR_SUCCESS) {
> #else
> - while ((rv = apr_getopt(opt, "lL:p:ftve", &c, &opt_arg)) ==
> APR_SUCCESS) {
> + while ((rv = apr_getopt(opt, "lL:p:i:ftve", &c, &opt_arg)) ==
> APR_SUCCESS) {
> #endif
> switch (c) {
> case 'l':
> @@ -543,6 +549,12 @@
> case 'p':
> config.postrotate_prog = opt_arg;
> break;
> + case 'i':
> + config.inputpipe = opt_arg;
> + break;
> + case 'o':
> + config.ignore_eof = 1;
> + break;
> case 'f':
> config.force_open = 1;
> break;
> @@ -588,9 +600,25 @@
>
> config.use_strftime = (strchr(config.szLogRoot, '%') != NULL);
>
> - if (apr_file_open_stdin(&f_stdin, status.pool) != APR_SUCCESS) {
> - fprintf(stderr, "Unable to open stdin\n");
> - exit(1);
> + if (config.inputpipe) {
> + rv = apr_file_namedpipe_open(&f_stdin, config.inputpipe,
> + APR_FOPEN_READ |
> + (config.ignore_eof ?
> APR_FOPEN_WRITE : 0),
> + APR_FULL_NONBLOCK,
> status.pool);
> +
> + if (rv != APR_SUCCESS) {
> + fprintf(stderr, "Unable to open pipe %s\n",
> config.inputpipe);
> + exit(1);
> + }
> + } else {
> + if (config.ignore_eof) {
> + usage(argv[0], "-o requires -i <inputpipe>");
> + }
> +
> + if (apr_file_open_stdin(&f_stdin, status.pool) !=
> APR_SUCCESS) {
> + fprintf(stderr, "Unable to open stdin\n");
> + exit(1);
> + }
> }
>
> if (apr_file_open_stdout(&f_stdout, status.pool) != APR_SUCCESS)
> {
> @@ -637,7 +665,11 @@
> if (pollret == APR_SUCCESS) {
> rv = apr_file_read(f_stdin, buf, &nRead);
> if (APR_STATUS_IS_EOF(rv)) {
> - break;
> + if (config.ignore_eof) {
> + continue;
> + } else {
> + break;
> + }
> }
> else if (rv != APR_SUCCESS) {
> exit(3);
> ===================================================================
>
> --
> GPG Key ID: 088335A9
> GPG Key Fingerprint: 86CD 3297 7493 75BC F820 6715 F54F E648 0883
> 35A9
>
>
--
Igor Galić
Tel: +43 (0) 664 886 22 883
Mail: i.galic@brainsware.org
URL: http://brainsware.org/
GPG: 6880 4155 74BD FD7C B515 2EA5 4B1D 9E08 A097 C9AE