You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Bojan Smojver <bo...@rexursive.com> on 2002/09/20 00:25:17 UTC
[PATCH]: mod_log_config.c
This time a version that actually compiles :-)
I have tested this and it seems to be working OK for both SSL and plain
connections. The patch also includes changes to the documentation and
sample configuration files, which can now take advantage of this
logging. I've called the new custom log "combinedio".
As usual, this is against the latest CVS head. I doubt that this will be
the last of it, but it might be a good start, or at least I hope ;-)
Finally, big thanks to everyone that helped me learn bits of Apache API
and for all the ideas.
Bojan
Re: [PATCH]: mod_log_config.c
Posted by Bojan Smojver <bo...@rexursive.com>.
Quoting Ian Holsman <ia...@apache.org>:
> I'm sorry for jumping in so late in the game, but
> why can't you do the same thing as a seperate module which implements
> the optional hook function to get those two directives in '%I/%O'
>
> i'm concerned that this will add undue burden on everybodies request
> wheather they use the %I/%O directives or not.
<sales-pitch>
I think this functionality is something that's been missing from Apache for a
long time, so I find it to be essential. It would also enable third party module
writers (e.g. mod_accounting and mod_watch) to rely on the fact that the
accurate numbers of input and output bytes per request are always recoreded,
given that mod_log_config is "very standard". The additions are also fairly
lightweight, thanks to suggestions by Ryan and Justin, so I don't think there
are going to be major performance implications. Furthermore, my sinister plan
involves writing a patch for webalizer if/when any of this becomes accepted into
Apache. And that would mean that the I/O numbers would be in widespread use in
no time at all :-)
</sales-pitch>
However, given that Apache has to make everyone happy (and I think that's a good
thing, not a bad thing), maybe we can compromise and have a configuration flag
(e.g. LogInputOutput on/off, default off), which will then be used in
logio_pre_conn() to determine if the filters should/should not be inserted. Does
that sound OK to you?
If no, I'm OK with anything that gives me this functionality. This whole thing
started of as mod_logio anyway...
Bojan
-------------------------------------------------
This mail sent through IMP: http://horde.org/imp/
Re: [PATCH]: mod_log_config.c
Posted by Ian Holsman <ia...@apache.org>.
Bojan Smojver wrote:
> This time a version that actually compiles :-)
>
> I have tested this and it seems to be working OK for both SSL and plain
> connections. The patch also includes changes to the documentation and
> sample configuration files, which can now take advantage of this
> logging. I've called the new custom log "combinedio".
>
> As usual, this is against the latest CVS head. I doubt that this will be
> the last of it, but it might be a good start, or at least I hope ;-)
>
> Finally, big thanks to everyone that helped me learn bits of Apache API
> and for all the ideas.
>
I'm sorry for jumping in so late in the game, but
why can't you do the same thing as a seperate module which implements
the optional hook function to get those two directives in '%I/%O'
i'm concerned that this will add undue burden on everybodies request
wheather they use the %I/%O directives or not.
> Bojan
>
>
> ------------------------------------------------------------------------
>
> diff -u --recursive --new-file httpd-2.0/docs/conf/httpd-nw.conf httpd-2.0-patched/docs/conf/httpd-nw.conf
> --- httpd-2.0/docs/conf/httpd-nw.conf Wed Sep 18 21:40:17 2002
> +++ httpd-2.0-patched/docs/conf/httpd-nw.conf Fri Sep 20 08:03:47 2002
> @@ -372,6 +372,7 @@
> # The following directives define some format nicknames for use with
> # a CustomLog directive (see below).
> #
> +LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
> LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
> LogFormat "%h %l %u %t \"%r\" %>s %b" common
> LogFormat "%{Referer}i -> %U" referer
> diff -u --recursive --new-file httpd-2.0/docs/conf/httpd-std.conf.in httpd-2.0-patched/docs/conf/httpd-std.conf.in
> --- httpd-2.0/docs/conf/httpd-std.conf.in Wed Sep 18 21:40:18 2002
> +++ httpd-2.0-patched/docs/conf/httpd-std.conf.in Fri Sep 20 08:03:18 2002
> @@ -474,6 +474,7 @@
> # The following directives define some format nicknames for use with
> # a CustomLog directive (see below).
> #
> +LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
> LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
> LogFormat "%h %l %u %t \"%r\" %>s %b" common
> LogFormat "%{Referer}i -> %U" referer
> diff -u --recursive --new-file httpd-2.0/docs/conf/httpd-win.conf httpd-2.0-patched/docs/conf/httpd-win.conf
> --- httpd-2.0/docs/conf/httpd-win.conf Wed Sep 18 21:40:18 2002
> +++ httpd-2.0-patched/docs/conf/httpd-win.conf Fri Sep 20 08:04:03 2002
> @@ -401,6 +401,7 @@
> # The following directives define some format nicknames for use with
> # a CustomLog directive (see below).
> #
> +LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
> LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
> LogFormat "%h %l %u %t \"%r\" %>s %b" common
> LogFormat "%{Referer}i -> %U" referer
> diff -u --recursive --new-file httpd-2.0/docs/manual/mod/mod_log_config.html.en httpd-2.0-patched/docs/manual/mod/mod_log_config.html.en
> --- httpd-2.0/docs/manual/mod/mod_log_config.html.en Fri Sep 13 10:15:25 2002
> +++ httpd-2.0-patched/docs/manual/mod/mod_log_config.html.en Fri Sep 20 08:05:39 2002
> @@ -138,6 +138,12 @@
> this conflicted with the historical ssl %...{var}c syntax.)</div>
> </td></tr>
>
> +<tr><td>%...I:</td>
> +<td>Bytes received, including request and headers, cannot be zero.</td></tr>
> +
> +<tr><td>%...O:</td>
> +<td>Bytes sent, including headers, cannot be zero.</td></tr>
> +
> </table>
>
> <p>The "..." can be nothing at all (<em>e.g.</em>, <code>"%h %u
> @@ -352,4 +358,4 @@
> TransferLog logs/access_log
> </code></p></div>
>
> -</div></div><div id="footer"><p class="apache">Maintained by the <a href="http://httpd.apache.org/docs-project/">Apache HTTP Server Documentation Project</a></p><p class="menu"><a href="../mod/">Modules</a> | <a href="../mod/directives.html">Directives</a> | <a href="../faq/">FAQ</a> | <a href="../glossary.html">Glossary</a> | <a href="../sitemap.html">Sitemap</a></p></div></body></html>
> \ No newline at end of file
> +</div></div><div id="footer"><p class="apache">Maintained by the <a href="http://httpd.apache.org/docs-project/">Apache HTTP Server Documentation Project</a></p><p class="menu"><a href="../mod/">Modules</a> | <a href="../mod/directives.html">Directives</a> | <a href="../faq/">FAQ</a> | <a href="../glossary.html">Glossary</a> | <a href="../sitemap.html">Sitemap</a></p></div></body></html>
> diff -u --recursive --new-file httpd-2.0/docs/manual/mod/mod_log_config.xml httpd-2.0-patched/docs/manual/mod/mod_log_config.xml
> --- httpd-2.0/docs/manual/mod/mod_log_config.xml Sat Sep 7 13:22:03 2002
> +++ httpd-2.0-patched/docs/manual/mod/mod_log_config.xml Fri Sep 20 08:05:37 2002
> @@ -143,6 +143,12 @@
> this conflicted with the historical ssl %...{var}c syntax.)</note>
> </td></tr>
>
> +<tr><td>%...I:</td>
> +<td>Bytes received, including request and headers, cannot be zero.</td></tr>
> +
> +<tr><td>%...O:</td>
> +<td>Bytes sent, including headers, cannot be zero.</td></tr>
> +
> </table>
>
> <p>The "..." can be nothing at all (<em>e.g.</em>, <code>"%h %u
> diff -u --recursive --new-file httpd-2.0/modules/loggers/mod_log_config.c httpd-2.0-patched/modules/loggers/mod_log_config.c
> --- httpd-2.0/modules/loggers/mod_log_config.c Mon Sep 16 23:17:57 2002
> +++ httpd-2.0-patched/modules/loggers/mod_log_config.c Fri Sep 20 07:57:32 2002
> @@ -153,9 +153,11 @@
> * 'X' = connection aborted before the response completed.
> * '+' = connection may be kept alive after the response is sent.
> * '-' = connection will be closed after the response is sent.
> - (This directive was %...c in late versions of Apache 1.3, but
> - this conflicted with the historical ssl %...{var}c syntax.)
> -*
> + * (This directive was %...c in late versions of Apache 1.3, but
> + * this conflicted with the historical ssl %...{var}c syntax.)
> + * %...I: bytes received, including request and headers, cannot be zero
> + * %...O: bytes sent, including headers, cannot be zero
> + *
> * The '...' can be nothing at all (e.g. "%h %u %r %s %b"), or it can
> * indicate conditions for inclusion of the item (which will cause it
> * to be replaced with '-' if the condition is not met). Note that
> @@ -236,6 +238,8 @@
> static ap_log_writer_init *log_writer_init = ap_default_log_writer_init;
> static int buffered_logs = 0; /* default unbuffered */
>
> +static const char logio_filter_name[] = "LOG_INPUT_OUTPUT";
> +
> /* POSIX.1 defines PIPE_BUF as the maximum number of bytes that is
> * guaranteed to be atomic when writing a pipe. And PIPE_BUF >= 512
> * is guaranteed. So we'll just guess 512 in the event the system
> @@ -298,6 +302,16 @@
> char *condition_var;
> } config_log_state;
>
> +/*****************************************************************
> + *
> + * Logging of input and output config ...
> + */
> +
> +typedef struct logio_config_t {
> + apr_off_t bytes_in;
> + apr_off_t bytes_out;
> +} logio_config_t;
> +
> /*
> * Format items...
> * Note that many of these could have ap_sprintfs replaced with static buffers.
> @@ -620,6 +634,22 @@
> return "-";
> }
>
> +static const char *log_bytes_in(request_rec *r, char *a)
> +{
> + logio_config_t *cf = ap_get_module_config(r->connection->conn_config,
> + &log_config_module);
> +
> + return apr_off_t_toa(r->pool, cf->bytes_in);
> +}
> +
> +static const char *log_bytes_out(request_rec *r, char *a)
> +{
> + logio_config_t *cf = ap_get_module_config(r->connection->conn_config,
> + &log_config_module);
> +
> + return apr_off_t_toa(r->pool, cf->bytes_out);
> +}
> +
> /*****************************************************************
> *
> * Parsing the log format string
> @@ -904,6 +934,8 @@
> {
> multi_log_state *mls = ap_get_module_config(r->server->module_config,
> &log_config_module);
> + logio_config_t *cf = ap_get_module_config(r->connection->conn_config,
> + &log_config_module);
> config_log_state *clsarray;
> int i;
>
> @@ -927,11 +959,62 @@
> }
> }
>
> + cf->bytes_in = cf->bytes_out = 0;
> +
> return OK;
> }
>
> /*****************************************************************
> *
> + * Logging of input and output filters ...
> + */
> +
> +static apr_status_t logio_out_filter(ap_filter_t *f,
> + apr_bucket_brigade *bb) {
> + apr_off_t length;
> + logio_config_t *cf = ap_get_module_config(f->c->conn_config,
> + &log_config_module);
> +
> + if (!cf) { /* Create config */
> + cf = apr_pcalloc(f->c->pool, sizeof(*cf));
> + ap_set_module_config(f->c->conn_config, &log_config_module, cf);
> + }
> +
> + apr_brigade_length (bb, 0, &length);
> +
> + if (length > 0)
> + cf->bytes_out += length;
> +
> + return ap_pass_brigade(f->next, bb);
> +}
> +
> +static apr_status_t logio_in_filter(ap_filter_t *f,
> + apr_bucket_brigade *bb,
> + ap_input_mode_t mode,
> + apr_read_type_e block,
> + apr_off_t readbytes) {
> + apr_off_t length;
> + apr_status_t status;
> + logio_config_t *cf = ap_get_module_config(f->c->conn_config,
> + &log_config_module);
> +
> + status = ap_get_brigade(f->next, bb, mode, block, readbytes);
> +
> + if (!cf) { /* Create config */
> + cf = apr_pcalloc(f->c->pool, sizeof(*cf));
> + ap_set_module_config(f->c->conn_config, &log_config_module, cf);
> + }
> +
> + apr_brigade_length (bb, 0, &length);
> +
> + if (length > 0)
> + cf->bytes_in += length;
> +
> + return status;
> +}
> +
> +/*****************************************************************
> + *
> * Module glue...
> */
>
> @@ -1333,6 +1416,13 @@
> return rv;
> }
>
> +static int logio_pre_conn(conn_rec *c) {
> + ap_add_input_filter(logio_filter_name, NULL, NULL, c);
> + ap_add_output_filter(logio_filter_name, NULL, NULL, c);
> +
> + return OK;
> +}
> +
> static int log_pre_config(apr_pool_t *p, apr_pool_t *plog, apr_pool_t *ptemp)
> {
> static APR_OPTIONAL_FN_TYPE(ap_register_log_handler) *log_pfn_register;
> @@ -1367,6 +1457,8 @@
> log_pfn_register(p, "T", log_request_duration, 1);
> log_pfn_register(p, "U", log_request_uri, 1);
> log_pfn_register(p, "s", log_status, 1);
> + log_pfn_register(p, "I", log_bytes_in, 0);
> + log_pfn_register(p, "O", log_bytes_out, 0);
> }
>
> return OK;
> @@ -1374,6 +1466,7 @@
>
> static void register_hooks(apr_pool_t *p)
> {
> + ap_hook_pre_connection(logio_pre_conn, NULL, NULL, APR_HOOK_MIDDLE);
> ap_hook_pre_config(log_pre_config,NULL,NULL,APR_HOOK_REALLY_FIRST);
> ap_hook_child_init(init_child,NULL,NULL,APR_HOOK_MIDDLE);
> ap_hook_open_logs(init_config_log,NULL,NULL,APR_HOOK_MIDDLE);
> @@ -1389,6 +1482,11 @@
> APR_REGISTER_OPTIONAL_FN(ap_register_log_handler);
> APR_REGISTER_OPTIONAL_FN(ap_log_set_writer_init);
> APR_REGISTER_OPTIONAL_FN(ap_log_set_writer);
> +
> + ap_register_input_filter(logio_filter_name, logio_in_filter, NULL,
> + AP_FTYPE_NETWORK - 1);
> + ap_register_output_filter(logio_filter_name, logio_out_filter, NULL,
> + AP_FTYPE_NETWORK - 1);
> }
>
> module AP_MODULE_DECLARE_DATA log_config_module =