You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Aidan Cully <ai...@panix.com> on 1998/12/12 01:58:43 UTC
adding vhost-specific data to shared error logs
Hello again..
I've attached a 300 line patch to apache which should allow what the
subject line says, though I haven't tested it yet. Obviously, I'll test
it before I even want it to go in; this message is just to get feedback
on how I've implemented what I want to do, and if anyone else is
interested in this. Basically, I've added an
AddLogger <name> /path/to/logger directive to httpd.conf, and made
ErrorLog %<name>:<string> talk to the logger named <name>. The logs that
get sent to <name> look like "<string> rest of log_error output", without
the quotes or the angle brackets (unless the string has angle brackets in
it). I was going to use it to add the following to our httpd.conf
AddLogger vhost "|/apache/bin/log-prog"
and this to all our virtual-hosts
ErrorLog "%vhost:UID GID /path/to/docroot/error_logs/errlog"
so that log-prog could securely create (if it didn't exist) and write to
/path/to/docroot/error_logs/errlog as the userid.
What do y'all think?
--aidan
--
Aidan Cully "I assume you're all in bands.."
Panix Staff --Robyn Hitchcock
aidan@panix.com
Re: adding vhost-specific data to shared error logs
Posted by Aidan Cully <ai...@panix.com>.
On Mon, Mar 01, 1999 at 07:46:20AM, Dean Gaudet said:
> replying to an old message...
>
> On Fri, 18 Dec 1998, Aidan Cully wrote:
> > shared-logger stuff
>
> It's not obvious to me why you're putting in this complex data structure.
> Wouldn't it be sufficient to use s->server_name as the "string" in pretty
> much all cases?
>
> This gets back to my KISS philosophy -- you're putting in all this extra
> logging gear inside apache when you could easily handle it outside apache
> in a piped logger. If what you're really looking for is the ability to
> send log messages to both syslog and a file, an external logger can handle
> that no problem. As long as the server_name is there as the first field,
> the logger can probably do what it wants.
>
> Oh, except for the dynamic vhost situation, which your proposal doesn't
> handle either.
I've changed my mind on this, and now I agree with Dean. *Teehee*!
I'm going to write my logger to use an external config file, not in any
way connected with Apache, that will map server_name to log-file/UID.
Someone else's logger could be made to work with dynamic vhosts (arseuming
those are what I think they are).
Now the question is, is there any reason not to just change the format
of all ErrorLogs to prepend the server_name to the format string?
I know I haven't spent any time on this at all in the past several
months.. I've had "other stuff" to work on, but upgrading our local
Apache is the next thing on my plate, now, so I should have a patch
ready real soon.
--aidan
--
Aidan Cully "Billy's mother was enormous. I looked at her, then looked
Panix Staff at the trailer door, than back at her, and I was faced
aidan@panix.com with my first real math problem." -- Tom Waits
Re: adding vhost-specific data to shared error logs
Posted by Aidan Cully <ai...@panix.com>.
On Mon, Mar 01, 1999 at 07:46:20AM, Dean Gaudet said:
> replying to an old message...
>
> On Fri, 18 Dec 1998, Aidan Cully wrote:
> > On Wed, Dec 16, 1998 at 04:57:59PM, Dean Gaudet said:
> > > On Tue, 15 Dec 1998, Aidan Cully wrote:
> > >
> > > > I hate to follow up to myself, but no one else has, and I really don't
> > > > want to continue wasting my time on this feature if no one cares enough
> > > > about it to let me know.
> > > >
> > > > So is this useful to people? Should I be posting to another forum, or
> > > > should I file a PR on it? Should I just forget about this feature?
> > >
> > > I think it's useful but I haven't had a chance to look at your work...
> >
> > Groovy, that's all I really needed to hear. I'll fix the bugs I know
> > about, do some testing, and resubmit it.
> >
> > To save time, here's an overview of the patch:
> > Change server_rec->error_log from a FILE* to a log_type_rec*, where
> > log_type_rec is defined as follows:
> > struct log_type_rec {
> > int type;
> > void *data;
> > };
> > where "data" is type dependant, and type can be one of LOG_TYPE_SYSLOG,
> > LOG_TYPE_FILE and LOG_TYPE_SHARE. For SYSLOG, data is ignored, for
> > FILE, data is a FILE*, and for SHARE, data is a pointer to a structure:
> > struct shared_log_rec {
> > char *string;
> > log_type_rec *share;
> > };
> > (of course, everything that deals directly with error_log has been
> > modified to expect log_type_rec's, rather than FILE*'s.)
> >
> > To support shared loggers, I added another field to server_rec, named
> > (surprisingly) shared_loggers. shared_loggers is an array of named
> > log_type_rec's.. The ap_init_virtual_host function has been modified
> > to use the same 'shared_loggers' pointer among every host. The
> > AddLogger configuration checks context, then adds another shared_logger
> > to this array. The way it does this is to call a modified
> > "open_error_log" function (called ap_open_one_log) that returns a
> > log_type_rec*, and associates that with a string. The old
> > open_error_log() has been modified to take the return value of
> > ap_open_one_log() and stick it into server_rec->error_log.
> >
> > When ap_open_one_log() encounters a shared logger declaration, it
> > searches through server_rec->shared_loggers for the appropriate one,
> > and associates a shared_log_rec* with the associated log_type_rec*.
> >
> > The log_error_core() function has been made recursive.. When an
> > error_log is SHAREd, it prints the header into a buffer and calls
> > itself again.
> >
> > Hope this is all clear..
>
> It's not obvious to me why you're putting in this complex data structure.
> Wouldn't it be sufficient to use s->server_name as the "string" in pretty
> much all cases?
Not quite.. Not unless I also had the external logger know how to parse
the apache config files, and be able to figure out which particular vhost
in the config was associated with the server_name. If y'all wanted to
externalize the config-parsing bits from src/main/, I might be willing to
go this route.
> This gets back to my KISS philosophy -- you're putting in all this extra
> logging gear inside apache when you could easily handle it outside apache
> in a piped logger.
First, piped loggers don't work for error logs.. Error logs currently
have no way of getting vhost specific data.
Second, I put in the more advanced logging gear with the idea that we'd
eventually be able to support more advanced syslogging.. E.g., be able
to set different log facilities, or different log masks for different
vhosts. FILE*/No FILE* is not quite enough to do that.
> If what you're really looking for is the ability to
> send log messages to both syslog and a file, an external logger can handle
> that no problem. As long as the server_name is there as the first field,
> the logger can probably do what it wants.
That's not what I'm looking for.. I'm trying to (securely) store the
error logs under a vhost's docroot, and owned by the vhost's UID. I'm
trying to do slightly non-obvious log-splitting for the error-logs,
that's all.
> Oh, except for the dynamic vhost situation, which your proposal doesn't
> handle either.
Oh.. Hmm.. Well, externalizing the config-parsing as I suggest above,
and using the SERVER_NAME environment variable as the initial string
should work pretty well for dynamic vhosts.. (not that externalizing
the config-parsing is necessary for dynamic vhosts, but it is for what
I want to do.)
--aidan
--
Aidan Cully "I saw Judas carryin' the body/ Of John Wilkes Booth..
Panix Staff Down there by the train..."
aidan@panix.com -Johnny Cash
Re: Disabling configuration directives
Posted by Tony Finch <do...@dotat.at>.
Ben Hyde <bh...@pobox.com> wrote:
>Tony Finch (a man with so many dots) writes:
>...
>>Because UseCanonicalName can change this, and because customers can
>>put it in their .htaccess files, they can therefore confuse the log
>>system and e.g. get free bandwidth...
>
>I've sometimes thought that a configuration directive that
>disabled a named command would be a good thing. - ben
Indeed, and also finer control over what Options may be set would be
nice...
Tony.
--
f.a.n.finch dot@dotat.at fanf@demon.net
Disabling configuration directives
Posted by Ben Hyde <bh...@pobox.com>.
Tony Finch (a man with so many dots) writes:
...
>Because UseCanonicalName can change this, and because customers can
>put it in their .htaccess files, they can therefore confuse the log
>system and e.g. get free bandwidth...
I've sometimes thought that a configuration directive that
disabled a named command would be a good thing. - ben
Re: adding vhost-specific data to shared error logs
Posted by Dean Gaudet <dg...@arctic.org>.
replying to an old message...
On Fri, 18 Dec 1998, Aidan Cully wrote:
> On Wed, Dec 16, 1998 at 04:57:59PM, Dean Gaudet said:
> > On Tue, 15 Dec 1998, Aidan Cully wrote:
> >
> > > I hate to follow up to myself, but no one else has, and I really don't
> > > want to continue wasting my time on this feature if no one cares enough
> > > about it to let me know.
> > >
> > > So is this useful to people? Should I be posting to another forum, or
> > > should I file a PR on it? Should I just forget about this feature?
> >
> > I think it's useful but I haven't had a chance to look at your work...
>
> Groovy, that's all I really needed to hear. I'll fix the bugs I know
> about, do some testing, and resubmit it.
>
> To save time, here's an overview of the patch:
> Change server_rec->error_log from a FILE* to a log_type_rec*, where
> log_type_rec is defined as follows:
> struct log_type_rec {
> int type;
> void *data;
> };
> where "data" is type dependant, and type can be one of LOG_TYPE_SYSLOG,
> LOG_TYPE_FILE and LOG_TYPE_SHARE. For SYSLOG, data is ignored, for
> FILE, data is a FILE*, and for SHARE, data is a pointer to a structure:
> struct shared_log_rec {
> char *string;
> log_type_rec *share;
> };
> (of course, everything that deals directly with error_log has been
> modified to expect log_type_rec's, rather than FILE*'s.)
>
> To support shared loggers, I added another field to server_rec, named
> (surprisingly) shared_loggers. shared_loggers is an array of named
> log_type_rec's.. The ap_init_virtual_host function has been modified
> to use the same 'shared_loggers' pointer among every host. The
> AddLogger configuration checks context, then adds another shared_logger
> to this array. The way it does this is to call a modified
> "open_error_log" function (called ap_open_one_log) that returns a
> log_type_rec*, and associates that with a string. The old
> open_error_log() has been modified to take the return value of
> ap_open_one_log() and stick it into server_rec->error_log.
>
> When ap_open_one_log() encounters a shared logger declaration, it
> searches through server_rec->shared_loggers for the appropriate one,
> and associates a shared_log_rec* with the associated log_type_rec*.
>
> The log_error_core() function has been made recursive.. When an
> error_log is SHAREd, it prints the header into a buffer and calls
> itself again.
>
> Hope this is all clear..
It's not obvious to me why you're putting in this complex data structure.
Wouldn't it be sufficient to use s->server_name as the "string" in pretty
much all cases?
This gets back to my KISS philosophy -- you're putting in all this extra
logging gear inside apache when you could easily handle it outside apache
in a piped logger. If what you're really looking for is the ability to
send log messages to both syslog and a file, an external logger can handle
that no problem. As long as the server_name is there as the first field,
the logger can probably do what it wants.
Oh, except for the dynamic vhost situation, which your proposal doesn't
handle either.
Dean
Re: Some logging issues & questions
Posted by Dean Gaudet <dg...@arctic.org>.
On Wed, 23 Dec 1998, Marc Slemko wrote:
> Except that you do need some way to log the hostname used if you are using
> it for non-vhost based vhosts with mod_rewrite. It is arguable that just
> using the Host: header there would be enough in most cases, although I
> think there may be a few where it isn't.
Hmm. Except %v and %p are defined as logging the canonical name/port, so
it was definately not working as documented... but maybe the docs are what
should be changed.
On Thu, 24 Dec 1998, Tony Finch wrote:
> I thought that might be the case. However, we are playing tricks with
> UseCanonicalName in order to avoid filling httpd.conf with thousands
> of <VirtualHost> configuration directives and having to kick the
> server when anything changes. This isn't a fix for us :-(
I'm starting to think that it's just wrong to let UseCanonicalName be
changed in .htaccess files.
Dean
Re: Some logging issues & questions
Posted by Tony Finch <do...@dotat.at>.
Dean Gaudet <dg...@arctic.org> wrote:
>On Wed, 23 Dec 1998, Tony Finch wrote:
>>
>> The latter step is done on the basis of what Apache puts in the %v
>> field, i.e. what it thinks the ServerName is when it writes the log.
>> Because UseCanonicalName can change this, and because customers can
>> put it in their .htaccess files, they can therefore confuse the log
>> system and e.g. get free bandwidth...
>
>Oh yuck, that wasn't intentional... I think mod_log_config should be
>changed to use r->server_hostname unconditionally... ditto for the
>port. I'll make that change.
I thought that might be the case. However, we are playing tricks with
UseCanonicalName in order to avoid filling httpd.conf with thousands
of <VirtualHost> configuration directives and having to kick the
server when anything changes. This isn't a fix for us :-(
Tony.
--
f.a.n.finch dot@dotat.at fanf@demon.net
Re: Some logging issues & questions
Posted by Marc Slemko <ma...@worldgate.com>.
On Wed, 23 Dec 1998, Dean Gaudet wrote:
> On Wed, 23 Dec 1998, Tony Finch wrote:
>
> > The latter step is done on the basis of what Apache puts in the %v
> > field, i.e. what it thinks the ServerName is when it writes the log.
> > Because UseCanonicalName can change this, and because customers can
> > put it in their .htaccess files, they can therefore confuse the log
> > system and e.g. get free bandwidth...
>
> Oh yuck, that wasn't intentional... I think mod_log_config should be
> changed to use r->server_hostname unconditionally... ditto for the
> port. I'll make that change.
Except that you do need some way to log the hostname used if you are using
it for non-vhost based vhosts with mod_rewrite. It is arguable that just
using the Host: header there would be enough in most cases, although I
think there may be a few where it isn't.
Re: Some logging issues & questions
Posted by Dean Gaudet <dg...@arctic.org>.
On Wed, 23 Dec 1998, Tony Finch wrote:
> The latter step is done on the basis of what Apache puts in the %v
> field, i.e. what it thinks the ServerName is when it writes the log.
> Because UseCanonicalName can change this, and because customers can
> put it in their .htaccess files, they can therefore confuse the log
> system and e.g. get free bandwidth...
Oh yuck, that wasn't intentional... I think mod_log_config should be
changed to use r->server_hostname unconditionally... ditto for the
port. I'll make that change.
Dean
Some logging issues & questions
Posted by Tony Finch <do...@dotat.at>.
I have a couple of questions about fiddling with the way Apache deals
with logs so that I can decide on the right direction in which to
hack...
On our current web server cluster we configure Apache to write access
logs to a fifo from which they are picked up by the log processing
system. This collates logs from the various front-end servers,
produces some stats for accounting purposes, and delivers the logs to
customers.
The latter step is done on the basis of what Apache puts in the %v
field, i.e. what it thinks the ServerName is when it writes the log.
Because UseCanonicalName can change this, and because customers can
put it in their .htaccess files, they can therefore confuse the log
system and e.g. get free bandwidth...
My current solution to this is to prevent them from using
UseCanonicalName by setting its req_override to RSRC_CONF which is OK,
but I want the best of both worlds :-) i.e. I want customers to be
able to fiddle with UseCanonicalName if they wish but I want to revoke
any config changes made by .htaccess files before the server gets
around to logging. Does anyone have any suggestions about how to do
this cleanly?
The other thing that caught my eye recently was Aidan Cully's error
log patch. I have been vaguely thinking about providing customers with
error logs via something similar to our system for access logs. A
quick hack to http_log.c would sort-of do the trick, or perhaps
something like mod_log_config (but that's probably overkill).
Any suggestions welcome.
Tony.
--
f.a.n.finch dot@dotat.at fanf@demon.net
Re: adding vhost-specific data to shared error logs
Posted by Aidan Cully <ai...@panix.com>.
On Wed, Dec 16, 1998 at 04:57:59PM, Dean Gaudet said:
> On Tue, 15 Dec 1998, Aidan Cully wrote:
>
> > I hate to follow up to myself, but no one else has, and I really don't
> > want to continue wasting my time on this feature if no one cares enough
> > about it to let me know.
> >
> > So is this useful to people? Should I be posting to another forum, or
> > should I file a PR on it? Should I just forget about this feature?
>
> I think it's useful but I haven't had a chance to look at your work...
Groovy, that's all I really needed to hear. I'll fix the bugs I know
about, do some testing, and resubmit it.
To save time, here's an overview of the patch:
Change server_rec->error_log from a FILE* to a log_type_rec*, where
log_type_rec is defined as follows:
struct log_type_rec {
int type;
void *data;
};
where "data" is type dependant, and type can be one of LOG_TYPE_SYSLOG,
LOG_TYPE_FILE and LOG_TYPE_SHARE. For SYSLOG, data is ignored, for
FILE, data is a FILE*, and for SHARE, data is a pointer to a structure:
struct shared_log_rec {
char *string;
log_type_rec *share;
};
(of course, everything that deals directly with error_log has been
modified to expect log_type_rec's, rather than FILE*'s.)
To support shared loggers, I added another field to server_rec, named
(surprisingly) shared_loggers. shared_loggers is an array of named
log_type_rec's.. The ap_init_virtual_host function has been modified
to use the same 'shared_loggers' pointer among every host. The
AddLogger configuration checks context, then adds another shared_logger
to this array. The way it does this is to call a modified
"open_error_log" function (called ap_open_one_log) that returns a
log_type_rec*, and associates that with a string. The old
open_error_log() has been modified to take the return value of
ap_open_one_log() and stick it into server_rec->error_log.
When ap_open_one_log() encounters a shared logger declaration, it
searches through server_rec->shared_loggers for the appropriate one,
and associates a shared_log_rec* with the associated log_type_rec*.
The log_error_core() function has been made recursive.. When an
error_log is SHAREd, it prints the header into a buffer and calls
itself again.
Hope this is all clear..
--aidan
--
Aidan Cully "I assume you're all in bands.."
Panix Staff --Robyn Hitchcock
aidan@panix.com
Re: adding vhost-specific data to shared error logs
Posted by Dean Gaudet <dg...@arctic.org>.
On Tue, 15 Dec 1998, Aidan Cully wrote:
> I hate to follow up to myself, but no one else has, and I really don't
> want to continue wasting my time on this feature if no one cares enough
> about it to let me know.
>
> So is this useful to people? Should I be posting to another forum, or
> should I file a PR on it? Should I just forget about this feature?
I think it's useful but I haven't had a chance to look at your work...
Dean
Re: adding vhost-specific data to shared error logs
Posted by Aidan Cully <ai...@panix.com>.
I hate to follow up to myself, but no one else has, and I really don't
want to continue wasting my time on this feature if no one cares enough
about it to let me know.
So is this useful to people? Should I be posting to another forum, or
should I file a PR on it? Should I just forget about this feature?
Anyone?
--aidan
On Fri, Dec 11, 1998 at 07:58:43PM, Aidan Cully said:
> Hello again..
> I've attached a 300 line patch to apache which should allow what the
> subject line says, though I haven't tested it yet. Obviously, I'll test
> it before I even want it to go in; this message is just to get feedback
> on how I've implemented what I want to do, and if anyone else is
> interested in this. Basically, I've added an
> AddLogger <name> /path/to/logger directive to httpd.conf, and made
> ErrorLog %<name>:<string> talk to the logger named <name>. The logs that
> get sent to <name> look like "<string> rest of log_error output", without
> the quotes or the angle brackets (unless the string has angle brackets in
> it). I was going to use it to add the following to our httpd.conf
> AddLogger vhost "|/apache/bin/log-prog"
> and this to all our virtual-hosts
> ErrorLog "%vhost:UID GID /path/to/docroot/error_logs/errlog"
> so that log-prog could securely create (if it didn't exist) and write to
> /path/to/docroot/error_logs/errlog as the userid.
--
Aidan Cully "I assume you're all in bands.."
Panix Staff --Robyn Hitchcock
aidan@panix.com