Envariables for logging phase only

[Rodent of Unusual Size <Ke...@Golux.Com>]

Someone brought up an interesting question on #apache: 'How can I log
everything *except* 404s?'

Obviously this can be done with piped logging, but I consider that a
bit of a handwave.  It appears that here really isn't any simple way
to accomplish what seems to actually be a reasonable thing.  It can be
worked around with ErrorDocument and mod_rewrite, I think, but that too
is a bag on the side.

So, unless someone has a solution I'm missing, I'd like to propose adding
a sort of pseudo-envariable during the logging phase.  Something like
'status_is_NNN', where NNN is appropriately substituted.  Then something
like

CustomLog logs/access_log combined env=!status_is_404

It doesn't immediately help the case of someone who wants to log (or not
log) some combination of status values, though, which is one reason I'm
just throwing this out as a straw man idea..
-- 
#ken	P-)}

Ken Coar, Sanagendamgagwedweinini  http://Golux.Com/coar/
Author, developer, opinionist      http://Apache-Server.Com/

"Millennium hand and shrimp!"

Re: Envariables for logging phase only

[Rodent of Unusual Size <Ke...@Golux.Com>]

James Cox wrote:
> 
> although, i am tending towards prefering a module for this; like,
> mod_log_on_steroids or something...

How do you plan to make the information available to other modules?
-- 
#ken	P-)}

Ken Coar, Sanagendamgagwedweinini  http://Golux.Com/coar/
Author, developer, opinionist      http://Apache-Server.Com/

"Millennium hand and shrimp!"

RE: Envariables for logging phase only

[James Cox <im...@php.net>]

>
> Rodent of Unusual Size wrote:
> >
> > Joshua Slive wrote:
> > >
> > > > LogStatus envname[=val] statusre ...
> > >
> > > (I think the envname should be last to align with SetEnvIf.)
> >
> > I didn't do it that way so that multiple status codes could be
> > specified, a la AddType and friends.
>
> But if the first argument is a regex, then switching them so
> the envariables are last, like SetEnvIf, is better.

Still, i think if we could make this named right it'd be better....
although, i am tending towards prefering a module for this; like,
mod_log_on_steroids or something...

 -- James

Re: Envariables for logging phase only

[Rodent of Unusual Size <Ke...@Golux.Com>]

Rodent of Unusual Size wrote:
> 
> Joshua Slive wrote:
> >
> > > LogStatus envname[=val] statusre ...
> >
> > (I think the envname should be last to align with SetEnvIf.)
> 
> I didn't do it that way so that multiple status codes could be
> specified, a la AddType and friends.

But if the first argument is a regex, then switching them so
the envariables are last, like SetEnvIf, is better.
-- 
#ken	P-)}

Ken Coar, Sanagendamgagwedweinini  http://Golux.Com/coar/
Author, developer, opinionist      http://Apache-Server.Com/

"Millennium hand and shrimp!"

Re: Envariables for logging phase only

[Joshua Slive <jo...@slive.ca>]

On Wed, 24 Jul 2002, Rodent of Unusual Size wrote:
> Joshua Slive wrote:
> > LogVariable %s "40[0-9]" exclude
> Fine and cool -- except that this is mod_log_config specific, and it
> would be nice to have something that was module-neutral.  Or are the
> logging format effectors dissociated from mod_log_config and moved into
> the core now?  I haven't kept up there..

I believe that in 2.0 it would be fairly easy to build a
mod_log_conditional that could provide this directive and use all the
functions from mod_log_config to implement it.  But I'm not planning to
write that myself, so the discussion is academic from my perspective.

Joshua.

Re: Envariables for logging phase only

[Rodent of Unusual Size <Ke...@Golux.Com>]

Joshua Slive wrote:
> 
> On Tue, 23 Jul 2002, Rodent of Unusual Size wrote:
> >
> > I personally find that a bit baroque. :-)  Also, it's tied
> > specifically to mod_log_config.  However, you've given me
> > an idea, so here's a counterproposal:
> >
> > LogStatus envname[=val] statusre ...
> 
> (I think the envname should be last to align with SetEnvIf.)

I didn't do it that way so that multiple status codes could be
specified, a la AddType and friends.

> I'm still in "blue sky" mode, so feel free to ignore me (as usual ;-), but
> if you're going to do that, why not go all the way:
> 
> LogVariable %s "40[0-9]" exclude
> LogVariable %a "127\.0\.0\.1" exclude
> LogVariable %U "\.jpg" exclude
> LogVariable %{Referer}i "yahoo.com" exclude
> Customlog ... env=!exclude
> 
> Where the first argument is a log-format %-directive, the second is a
> regex, and the third is an environment variable.  I would think that this
> should be relatively easy to do given the way mod_log_config works in 2.0.

Fine and cool -- except that this is mod_log_config specific, and it
would be nice to have something that was module-neutral.  Or are the
logging format effectors dissociated from mod_log_config and moved into
the core now?  I haven't kept up there..

> And yes, I'm still a big proponent of not using conditional logging at
> all; log everything and postprocess to get rid of what you don't want.

If you could, would you have your mail server reject spam on detection?
Or continue to accept *everything* and filter out even known spam after
the fact?  (Actually, I think you *can* teach sendmail to reject spam
using milters..)  Regardless, there are those (and I'm among them) who
think that conditional logging is more efficient and the best way of
doing some things.  It's in there, so let's not have this philosophical
discussion again.. :-)
-- 
#ken	P-)}

Ken Coar, Sanagendamgagwedweinini  http://Golux.Com/coar/
Author, developer, opinionist      http://Apache-Server.Com/

"Millennium hand and shrimp!"

Re: apache freebsd port --enable-so

[Tony Finch <do...@dotat.at>]

On Tue, Jul 23, 2002 at 11:43:54PM +0100, Dev Zero G Ltd wrote:
> How can I make "make" compile/configure/make apache 2 with the 
> --enable-so option? I can not run ./configure since I already have a 
> Makefile.

If you are using the FreeBSD port then it does --enable-so by default.

Tony.
-- 
f.a.n.finch <do...@dotat.at> http://dotat.at/
DOGGER: WEST OR NORTHWEST 4 OR 5, OCCASIONALLY 6 AT FIRST. SHOWERS THEN RAIN.
GOOD BECOMING MODERATE.

apache freebsd port --enable-so

[Dev Zero G Ltd <ma...@devzerog.com>]

How can I make "make" compile/configure/make apache 2 with the 
--enable-so option? I can not run ./configure since I already have a 
Makefile.

Thanks for any help in advance.

Mike

RE: Envariables for logging phase only

[James Cox <im...@php.net>]

> (I think the envname should be last to align with SetEnvIf.)
>
> I'm still in "blue sky" mode, so feel free to ignore me (as usual ;-), but
> if you're going to do that, why not go all the way:
>
> LogVariable %s "40[0-9]" exclude
> LogVariable %a "127\.0\.0\.1" exclude
> LogVariable %U "\.jpg" exclude
> LogVariable %{Referer}i "yahoo.com" exclude
> Customlog ... env=!exclude
>
i was thinking more along the lines of simply being able to send data to
different files based upon status code.. but then, being able to do that
with logvariables.. hmm, that's interesting too...

Still, postprocessing is probably better. i mean, how can x lines of code
replace:

grep 404 access_log > foo

i'm not sure this needs to be in core.. hmm.. (hmm, hmm, ..)

 -- james

Re: Envariables for logging phase only

[Joshua Slive <jo...@slive.ca>]

On Tue, 23 Jul 2002, Rodent of Unusual Size wrote:
>
> I personally find that a bit baroque. :-)  Also, it's tied
> specifically to mod_log_config.  However, you've given me
> an idea, so here's a counterproposal:
>
> LogStatus envname[=val] statusre ...

(I think the envname should be last to align with SetEnvIf.)

I'm still in "blue sky" mode, so feel free to ignore me (as usual ;-), but
if you're going to do that, why not go all the way:

LogVariable %s "40[0-9]" exclude
LogVariable %a "127\.0\.0\.1" exclude
LogVariable %U "\.jpg" exclude
LogVariable %{Referer}i "yahoo.com" exclude
Customlog ... env=!exclude

Where the first argument is a log-format %-directive, the second is a
regex, and the third is an environment variable.  I would think that this
should be relatively easy to do given the way mod_log_config works in 2.0.
And it both offers more power than any existing exclusion method, plus it
seems much more obvious.  If you want to exclude anything with xyz from
the logs, it makes sense to use a similar notation to what is necessary to
get xyz to appear in the logs in the first place.

And yes, I'm still a big proponent of not using conditional logging at
all; log everything and postprocess to get rid of what you don't want.
But it is such a commonly requested feature...

Joshua.

Re: Envariables for logging phase only

[Rodent of Unusual Size <Ke...@Golux.Com>]

James Cox wrote:
> 
> thus, to not log any status but 404 (ie, for a broken link log)
> 
> LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\""
> brokenlinks
> LogStatus "404, !*" brokenlinks
> CustomLog /var/log/httpd/brokenlinks_log brokenlinks

I personally find that a bit baroque. :-)  Also, it's tied
specifically to mod_log_config.  However, you've given me
an idea, so here's a counterproposal:

LogStatus envname[=val] statusre ...

as in

LogStatus status_is_redirect "^3.."
LogStatus status_is_success 200 206 304
LogStatus status_is_notfound 404
CustomLog logs/broken-links.log combined env=status_is_notfound

This would be handled by the core, and any appropriate envariables
would be set before any of the logging-phase hooks were called.
This way we make use of the existing conditional logging syntax,
the envariable concept, and make the facility available to other
logging modules than mod_log_config.
-- 
#ken	P-)}

Ken Coar, Sanagendamgagwedweinini  http://Golux.Com/coar/
Author, developer, opinionist      http://Apache-Server.Com/

"Millennium hand and shrimp!"

RE: Envariables for logging phase only

[James Cox <im...@php.net>]

> CustomLog logs/access_log combined env=!status_is_404
>
I'm not entirely sure about what the rules on adding new conf options are...
but,
what about using the logformat rules, and saying something like:

LogStatus "<combination of status codes, with ! used to not log them>"
logname (eg, combined).

This would also support the wildcard, so if it didn't know what to do for
that status (ie, that status was not in the list) it would perform the
default action.

If the user didn't express a log format, then the default would be :

LogStatus "*" combined

thus, to not log any status but 404 (ie, for a broken link log)

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\""
brokenlinks
LogStatus "404, !*" brokenlinks
CustomLog /var/log/httpd/brokenlinks_log brokenlinks

i might be interested in having a crack at this.

 -- James

Re: Envariables for logging phase only

["William A. Rowe, Jr." <wr...@rowe-clan.net>]

At 10:21 AM 7/22/2002, Rodent of Unusual Size wrote:

>Obviously this can be done with piped logging, but I consider that a
>bit of a handwave.  It appears that here really isn't any simple way
>to accomplish what seems to actually be a reasonable thing.  It can be
>worked around with ErrorDocument and mod_rewrite, I think, but that too
>is a bag on the side.
>
>So, unless someone has a solution I'm missing, I'd like to propose adding
>a sort of pseudo-envariable during the logging phase.  Something like
>'status_is_NNN', where NNN is appropriately substituted.  Then something
>like
>
>CustomLog logs/access_log combined env=!status_is_404

Wouldn't it make more sense to bite the bullet, skip the extra envvar
creation, and finally add status as an alternative, where we provide env?

I'd rather we err on the side of feature richness, then to start proliferating
more envvars that must be set up on every request [even when they
aren't used.]

Bill

Re: Envariables for logging phase only

[Joshua Slive <jo...@slive.ca>]

On Mon, 22 Jul 2002, Rodent of Unusual Size wrote:
> So, unless someone has a solution I'm missing, I'd like to propose adding
> a sort of pseudo-envariable during the logging phase.  Something like
> 'status_is_NNN', where NNN is appropriately substituted.  Then something
> like
>
> CustomLog logs/access_log combined env=!status_is_404
>
> It doesn't immediately help the case of someone who wants to log (or not
> log) some combination of status values, though, which is one reason I'm
> just throwing this out as a straw man idea..

In an ideal world, mod_setenvif would have access to a STATUS variable
that would give the reponse status and allow things like

SetEnvIf STATUS (404|401|403) failure
CustomLog logs/access_log combined env=!failure

And then, of course, people would also be able to do absurd things like
SetEnvIf STATUS 206 partial-content
Order allow,deny
allow from all
deny from env=partial-content
(Don't ask me why someone would want to do that, but similar things
do come up.)

Of course, I believe all these things are completely impossible,
because mod_setenvif hooks in way too early to know the status.

mod_rewrite actually comes close to having this capability using the
RewriteCond -U switch to do a sub-request.  But it doesn't offer access
to the actual status code, and it is probably too slow to use in general.
I believe it could actually work for the failure/not-failure case:
RewriteCond %{REQUEST_URI} -U
RewriteRule .* - [E=success]
CustomLog logs/access_log combined env=success
but that is slow and ugly.

Anyway, I'm done speculating on what things would look like in the world
of perfect software.

Joshua.