You are viewing a plain text version of this content. The canonical link for it is here.
Posted to apache-bugdb@apache.org by Lenny Foner <fo...@media.mit.edu> on 1999/08/03 20:02:55 UTC
general/4809: It would be nice if Apache could write compressed logfiles.
>Number: 4809
>Category: general
>Synopsis: It would be nice if Apache could write compressed logfiles.
>Confidential: no
>Severity: non-critical
>Priority: medium
>Responsible: apache
>State: open
>Class: change-request
>Submitter-Id: apache
>Arrival-Date: Tue Aug 3 11:10:01 PDT 1999
>Last-Modified:
>Originator: foner@media.mit.edu
>Organization:
apache
>Release: 1.3
>Environment:
Currently HPUX 10.20/gcc, but this is an
architecture-independent feature request.
(I currently administrate an Apache server
that serves about 1/3 of the Media Lab.)
>Description:
Apache currently writes uncompressed logs everywhere, which take up
lots of room and are much slower to serve over NFS than compressed logs
would be. We routinely compress our logs when they are rotated, and
searching them with gzip can be 10x faster because gunzip on a modern
CPU can be far faster than NFS on most LANs---the fact that so much less
data need cross the network more than makes up for the overhead of doing
the decompression. Further, the compression saves a tremendous amount
of space---most logfiles are highly compressible. Keeping them compressed
while written means that sites would not have to have enough logfile space
for the log before it gets rotated and -then- compressed, for example.
>How-To-Repeat:
Look at any Apache's logfiles.
>Fix:
Build the gzip library into Apache; it's already available in autoconf'ed
form and so forth and is used in many GNU projects. (You should of course
check, but I believe it's covered by the LGPL, which should mean you can
use it without fear of infection even though Apache itself doesn't appear
to be GPL'ed.)
Given that, it should be a relatively simple matter to stream logging output
into the compressor subroutines, and then write -those- to disk. You'd have
to provide configuration options that tell whether to do this on a per-logfile
(probably) basis, and probably also the compression level desired. Webmasters
would also need to understand that the buffering imposed by the compression
library means that the latency between a request and seeing it in the compressed
output would be significantly larger than with uncompressed logs, of course.
I've used the gzip libraries in projects of mine and could conceivably offer
some help, but I have -not- looked at the innards of the Apache logging code,
so that help might be somewhat limited. It's an easy library to use, however.
>Audit-Trail:
>Unformatted:
[In order for any reply to be added to the PR database, you need]
[to include <ap...@Apache.Org> in the Cc line and make sure the]
[subject line starts with the report component and number, with ]
[or without any 'Re:' prefixes (such as "general/1098:" or ]
["Re: general/1098:"). If the subject doesn't match this ]
[pattern, your message will be misfiled and ignored. The ]
["apbugs" address is not added to the Cc line of messages from ]
[the database automatically because of the potential for mail ]
[loops. If you do not include this Cc, your reply may be ig- ]
[nored unless you are responding to an explicit request from a ]
[developer. Reply only with text; DO NOT SEND ATTACHMENTS! ]
Re: general/4809: It would be nice if Apache could write compressed
logfiles.
Posted by Dirk-Willem van Gulik <di...@webweaving.org>.
On Wed, 4 Aug 1999, Manoj Kasichainula wrote:
> On Wed, Aug 04, 1999 at 12:39:07PM +0200, Dirk-Willem van Gulik wrote:
> >
> > > >Synopsis: It would be nice if Apache could write compressed logfiles.
>
> What's wrong with using piped logs in this case?
Nothing, pure 'feature' hunting.
I've recently started to appreciate more what it takes to roll out apache
in a environment where your average adiministrator is rented from
companies like origin or cmg. And that means that they propably do not
know of pipes (or vi; I had to install 'ee' for editing the password
files).
And then being able to select one from a commercial vendor which does all
this by magic (or as part of the binary distribution) is an asset. Not in
the least also because you can then tick another checkbox on the ISM or IT
managers list. Who usually is an accountant whou counts anyway.
Plus I still like the idea to have, everywhere where we reference a
'stream' or 'file' to use a URI or URL. Esp. as the proxy already needs
to know a few thing about that anyway.
Dw.
Re: general/4809: It would be nice if Apache could write compressed logfiles.
Posted by Manoj Kasichainula <ma...@io.com>.
On Wed, Aug 04, 1999 at 12:39:07PM +0200, Dirk-Willem van Gulik wrote:
>
> > >Synopsis: It would be nice if Apache could write compressed logfiles.
What's wrong with using piped logs in this case?
--
Manoj Kasichainula - manojk at io dot com - http://www.io.com/~manojk/
Re: general/4809: It would be nice if Apache could write
compressedlogfiles.
Posted by Rasmus Lerdorf <ra...@raleigh.ibm.com>.
> > > Doing so would also allow us to do the compression-on-the-fly
> > > content delivery people have requested. No?
> >
> > Correct. zlib gives you all the standard file io functions like gzopen,
> > gzclose, gzread, gzwrite, gzeof, etc.
> >
> > -Rasmus
>
> So, how would this be done though? A large site doing a lot of hits/sec
> could end up spending most of its time just compressing log files.
Hey, I was just relaying information. I think it is a terrible idea to
make Apache write compressed log files on the fly.
-Rasmus
Re: general/4809: It would be nice if Apache could write
compressedlogfiles.
Posted by Vincent Janelle <vj...@home.com>.
Rasmus Lerdorf wrote:
>
> > > Trivial to do by linking in zlib.
> >
> > Doing so would also allow us to do the compression-on-the-fly
> > content delivery people have requested. No?
>
> Correct. zlib gives you all the standard file io functions like gzopen,
> gzclose, gzread, gzwrite, gzeof, etc.
>
> -Rasmus
So, how would this be done though? A large site doing a lot of hits/sec
could end up spending most of its time just compressing log files.
I still like the idea of a logging process, perhaps something that could
be centerally managed, but I should probably try to write one before
asking for one =)
--
------------
The typical Unix hacker never can remember what the PRINT command is
called
this week -- but when it gets right down to it, Unix is a glorified
video
game. People don't do Serious Work on Unix systems: they send jokes
around
the world on UUCP-net and write adventure games and research papers.
--http://random.gimp.org --mailto:random@gimp.org --UIN 23939474
Re: general/4809: It would be nice if Apache could write compressed
logfiles.
Posted by Rasmus Lerdorf <ra...@apache.org>.
> > Trivial to do by linking in zlib.
>
> Doing so would also allow us to do the compression-on-the-fly
> content delivery people have requested. No?
Correct. zlib gives you all the standard file io functions like gzopen,
gzclose, gzread, gzwrite, gzeof, etc.
-Rasmus
Re: general/4809: It would be nice if Apache could write compressed
logfiles.
Posted by Rodent of Unusual Size <Ke...@Golux.Com>.
Rasmus Lerdorf wrote:
>
> > Agreed. I like this.
> >
> > I too see the benefit of compressing the log files directly rather
> > than the pipe option.
>
> Trivial to do by linking in zlib.
Doing so would also allow us to do the compression-on-the-fly
content delivery people have requested. No?
--
#ken P-)}
Ken Coar <http://Web.Golux.Com/coar/>
Apache Software Foundation <http://www.apache.org/>
"Apache Server for Dummies" <http://ASFD.MeepZor.Com/>
RE: general/4809: It would be nice if Apache could write compressed
logfiles.
Posted by Dirk-Willem van Gulik <di...@webweaving.org>.
On Thu, 5 Aug 1999, Rasmus Lerdorf wrote:
> > Agreed. I like this.
> >
> > I too see the benefit of compressing the log files directly rather
> > than the pipe option.
>
> Trivial to do by linking in zlib.
>
Yup. I am having a go at this (proxy hacking is no fun) and see if I can
get 'http' prefix hack for 1.2 ready for 2.0 in an easy to extend shape.
As we now have the hooks I am seriously thinking of making use of it non
http type modules; i.e. things which have nothing to do with handling
a request at all.
Two things are foremost on my mind; read/write modules; such as http:
socket: zip and 'table backend modules'; which sit behind the table
functions. At create time you can specify to do either flat, hashed or
even shared tables. Using appropriate backends. Obvious one are hashed
tables, SQL based persisten tables and cross process shared Berkeley DB
tables. That has been in my out tray long enough and should be done
soemthign with.
Dw
RE: general/4809: It would be nice if Apache could write compressed
logfiles.
Posted by Rasmus Lerdorf <ra...@apache.org>.
> Agreed. I like this.
>
> I too see the benefit of compressing the log files directly rather
> than the pipe option.
Trivial to do by linking in zlib.
-Rasmus
RE: general/4809: It would be nice if Apache could write compressed logfiles.
Posted by Dean Gaudet <dg...@arctic.org>.
On Thu, 5 Aug 1999, Brian Behlendorf wrote:
> On Thu, 5 Aug 1999, Randy Terbush wrote:
> > I too see the benefit of compressing the log files directly rather
> > than the pipe option.
>
> I see compression and (whether write to a file or a pipe) as independent
> options.
"Those who don't understand unix are doomed to re-invent it."
whose quote is that?
Dean
RE: general/4809: It would be nice if Apache could write compressed
logfiles.
Posted by Brian Behlendorf <br...@hyperreal.org>.
On Thu, 5 Aug 1999, Randy Terbush wrote:
> I too see the benefit of compressing the log files directly rather
> than the pipe option.
I see compression and (whether write to a file or a pipe) as independent
options.
Brian
RE: general/4809: It would be nice if Apache could write compressed logfiles.
Posted by Randy Terbush <ra...@covalent.net>.
Agreed. I like this.
I too see the benefit of compressing the log files directly rather
than the pipe option.
> -----Original Message-----
> From: new-httpd-owner@apache.org
> [mailto:new-httpd-owner@apache.org]On
> Behalf Of Rodent of Unusual Size
> Sent: Thursday, August 05, 1999 10:00 AM
> To: new-httpd@apache.org
> Subject: Re: general/4809: It would be nice if Apache could write
> compressed logfiles.
>
>
> Dirk-Willem van Gulik wrote:
> >
> > Looking at a lot of requests like this; perhaps in 2.0 it would
> > make sense to allow for _all_ places where we configure files
> > to have URI's listing them; i.e.
> >
> > file:/var/log/error_log-http
> > http://configs.swip.net/srm.conf
> > socket://loghost:1234
>
> I rather like this.. although it's turning the server into a
> client as well. :-) The lines blur..
> --
> #ken P-)}
>
> Ken Coar <http://Web.Golux.Com/coar/>
> Apache Software Foundation <http://www.apache.org/>
> "Apache Server for Dummies" <http://ASFD.MeepZor.Com/>
>
Re: general/4809: It would be nice if Apache could write compressed
logfiles.
Posted by Dirk-Willem van Gulik <di...@webweaving.org>.
y
On Thu, 5 Aug 1999, Rodent of Unusual Size wrote:
> Dirk-Willem van Gulik wrote:
> >
> > Looking at a lot of requests like this; perhaps in 2.0 it would
> > make sense to allow for _all_ places where we configure files
> > to have URI's listing them; i.e.
> >
> > file:/var/log/error_log-http
> > http://configs.swip.net/srm.conf
> > socket://loghost:1234
>
> I rather like this.. although it's turning the server into a
> client as well. :-) The lines blur..
>
Blame the proxy. I just could not resist doing the http:/ hack :-)
Dw.
Re: general/4809: It would be nice if Apache could write compressed
logfiles.
Posted by Rodent of Unusual Size <Ke...@Golux.Com>.
Dirk-Willem van Gulik wrote:
>
> Looking at a lot of requests like this; perhaps in 2.0 it would
> make sense to allow for _all_ places where we configure files
> to have URI's listing them; i.e.
>
> file:/var/log/error_log-http
> http://configs.swip.net/srm.conf
> socket://loghost:1234
I rather like this.. although it's turning the server into a
client as well. :-) The lines blur..
--
#ken P-)}
Ken Coar <http://Web.Golux.Com/coar/>
Apache Software Foundation <http://www.apache.org/>
"Apache Server for Dummies" <http://ASFD.MeepZor.Com/>
Re: general/4809: It would be nice if Apache could write compressed
logfiles.
Posted by Dirk-Willem van Gulik <di...@webweaving.org>.
> >Synopsis: It would be nice if Apache could write compressed logfiles.
Looking at a lot of requests like this; perhaps in 2.0 it would make sense
to allow for _all_ places where we configure files to have URI's listing
them; i.e.
file:/var/log/error_log-http
http://configs.swip.net/srm.conf
socket://loghost:1234
...
And then be able to handle the open/read/write/close/... in a modular
fashion so you could slot in a gzip:/var/log.. module.
And before you know it you have re-invented unix pipes. This was actually
what stopped me when I put in http: and socket: in 1.3 for a customer.
Hmm.
DW.