You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Greg Stein <gs...@lyra.org> on 1999/11/12 01:39:48 UTC
axe proxy support from 2.0?
This came up a while back, and there is actually an item in the STATUS
file, apparently (Manoj said so :-), but why do we still have the proxy
support in Apache? It seems like it has a largish number of bugs and is
pretty orthogonal to the rest of Apache. Can't we just axe it and refer
people to Squid?
It seems like there are sufficient alternatives in the open-source world
that we shouldn't try to pretend to be proxy developers...
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
Re: axe proxy support from 2.0?
Posted by Martin Kraemer <Ma...@mch.sni.de>.
On Sat, Nov 13, 1999 at 05:41:23PM -0000, David Reid wrote:
> I think the point that Gregs been making (and he'll correct me if I'm wrong
> I'm sure) is that what most people want is the functionality of various bits
> of a proxy. Given this it seems that if we concentrate on writing those
> bits to be tightly integrated with Apache-2.0 and abandon the idea that we
> supply a "full-blown" proxy we're getting more bang for our buck.
Only if you mean to say that "what most people want" is "what most
developers want" (like developers for mod_dav, mod_perl, mod_php and
the like). When we talk about "what most users want" (i.e., the
people who simply USE apache without doing active development)
then the picture could look quite different. I am quite sure that
among the apache USERS there is a large amount who use the apache+
proxy combo, especially because it's well suited for company
intranet use, and who would be quite disappointed it we simply
ditched it. I, for example, am such a "happy customer".
I know about most of the deficiencies of the current proxy, and
could probably hack squid to run on our BS2000 mainframes, but the
apache solution does quite easily and successfully today (what I
would have to work hard for to get it in the future if the proxy was
axed). And I think other apache users are in a similar situation.
After the discussion however, I think that the *perfect* solution
was really to
a) extend the API to get the "gimme that URL's contents"
functionality, plus
b) add the caching lib to store results in (and do it in a way so
that the request dimensions mime type, encoding, language etc. are
stored as well; see RFC2616 for details on HTTP/1.1 caching)
and then:
c) write a mod_proxy2 based on this functionality. It would become
quite simple, because many stupid things which happen today in
mod_proxy are based on problems with a) and b).
Until we actually _do_ have a), b) and c) we serve our users better
by porting the current mod_proxy.
And no: ftp is not dead yet.
Just my $.02
Martin
--
<Ma...@MchP.Siemens.De> | Fujitsu Siemens
Fon: +49-89-636-46021, FAX: +49-89-636-41143 | 81730 Munich, Germany
Re: axe proxy support from 2.0?
Posted by Martin Kraemer <Ma...@mch.sni.de>.
On Wed, Nov 17, 1999 at 09:15:11AM -0500, Ryan Bloom wrote:
>
> I HATE to volunteer for things, but if nobody else is willing to maintain
> mod_proxy, I will. :-(
>
> I agree we need it for 2.0, and that it is badly in need of a re-write. I
> don't have the time to re-write it, but I can keep the 1.3 proxy up and
> working for 2.0.
If I find the time, I'll help.
Martin
--
<Ma...@MchP.Siemens.De> | Fujitsu Siemens
Fon: +49-89-636-46021, FAX: +49-89-636-41143 | 81730 Munich, Germany
Re: axe proxy support from 2.0?
Posted by Ryan Bloom <rb...@raleigh.ibm.com>.
I HATE to volunteer for things, but if nobody else is willing to maintain
mod_proxy, I will. :-(
I agree we need it for 2.0, and that it is badly in need of a re-write. I
don't have the time to re-write it, but I can keep the 1.3 proxy up and
working for 2.0.
Ryan
On Wed, 17 Nov 1999, Martin Kraemer wrote:
> On Sat, Nov 13, 1999 at 11:45:40AM -0800, Greg Stein wrote:
> >
> > Gotta dig back thru my mail... I forget who has been maintaining the proxy
> > module. Maybe they're up for moving all/some of it to 2.0.
>
> I've been diggin' thru it a coupla times. But when I started, it was
> still Chuck Murcko's baby.
>
> Martin
> --
> <Ma...@MchP.Siemens.De> | Fujitsu Siemens
> Fon: +49-89-636-46021, FAX: +49-89-636-41143 | 81730 Munich, Germany
>
_______________________________________________________________________
Ryan Bloom rbb@raleigh.ibm.com
4205 S Miami Blvd
RTP, NC 27709 It's a beautiful sight to see good dancers
doing simple steps. It's a painful sight to
see beginners doing complicated patterns.
Re: axe proxy support from 2.0?
Posted by Martin Kraemer <Ma...@mch.sni.de>.
On Sat, Nov 13, 1999 at 11:45:40AM -0800, Greg Stein wrote:
>
> Gotta dig back thru my mail... I forget who has been maintaining the proxy
> module. Maybe they're up for moving all/some of it to 2.0.
I've been diggin' thru it a coupla times. But when I started, it was
still Chuck Murcko's baby.
Martin
--
<Ma...@MchP.Siemens.De> | Fujitsu Siemens
Fon: +49-89-636-46021, FAX: +49-89-636-41143 | 81730 Munich, Germany
Re: axe proxy support from 2.0?
Posted by Marc Slemko <ma...@znep.com>.
I'm not sure that I can agree with the suggestion that various random bits
of proxy-like functionality should be thrown in as someone feels like it
then, maybe, down the line someone may try to piece them all together.
I really do think you would get a much more useful set of APIs if you
designed them around a proxy. There are numerous issues that a simple
"gimme this document" doesn't have to worry about, but to add any caching
to that you really start getting all the functionality you need for a
proxy, so you may as well just do it.
Re: axe proxy support from 2.0?
Posted by Graham Leggett <mi...@sharp.fm>.
Greg Stein wrote:
> The second issue, which actually underlies some of the "bang for our buck"
> is that the proxy stuff has not been ported to the 2.0 framework yet.
> Nobody seems ready to step up and do that, so I recommended to just kill
> the thing. Feedback is against that, though :-), so it looks like we have
> a trimming rather than an axe-ing (how the heck do you spell that? :-)
I think it's less a trimming and more a reorganisation.
The proxy module is made up of some useful bits that could be even more
useful if it were possible to use them elsewhere. The caching code for
example could replace the functionality provided by mod_mmap_static,
providing a full blown cache for all requests, including normal
filesystem requests as well as remote requests. An available HTTP client
library would also be really useful, and would make patching in HTTPS
support behind a proxy a lot easier (if designed correctly).
I wouldn't mind stepping forward to help on some of this stuff, however
I have not yet hacked inside v2.0 yet. If you need someone to test any
of the code, I can give it a bash.
Regards,
Graham
--
-----------------------------------------
minfrin@sharp.fm "There's a moon
over Bourbon Street
tonight...
Re: axe proxy support from 2.0?
Posted by Greg Stein <gs...@lyra.org>.
On Sat, 13 Nov 1999, Eli Marmor wrote:
> David Reid wrote:
> > I think the point that Gregs been making (and he'll correct me if I'm wrong
> > I'm sure) is that what most people want is the functionality of various bits
> > of a proxy. Given this it seems that if we concentrate on writing those
> > bits to be tightly integrated with Apache-2.0 and abandon the idea that we
> > supply a "full-blown" proxy we're getting more bang for our buck.
Bing! Well-said.
The second issue, which actually underlies some of the "bang for our buck"
is that the proxy stuff has not been ported to the 2.0 framework yet.
Nobody seems ready to step up and do that, so I recommended to just kill
the thing. Feedback is against that, though :-), so it looks like we have
a trimming rather than an axe-ing (how the heck do you spell that? :-)
> My message was not a response to Greg's message (BTW: I think that he's
> doing some amazing projects, like the mod_dav and other useful things);
Thanx! :-)
> It was a response to the whole thread/idea, and his message was only the
> last one I saw when I decided to write my message. In any case, I under-
> stood the point that you are trying to clarify, when I wrote the
> previous message, and still thought that many people would miss that
> stuff. Again, I have good reasons for my opinion, but no time and/or
> desire to argue about them. I think that the fact that many people rely
> on things which are not among the "various bits" you mentioned, is
> enough to leave the whole proxy module as is, and logical reasons are
> not needed.
Your opinion is always valuable, developer or not. We do like to
concentrate on things that specifically interest us, but (speaking for
myself at least) we also like to know what people want from Apache.
While I can understand that you (and I'm sure there are others) would like
to see the proxy completely retained, it may not be possible unless
somebody wants to champion it and bring it up to 2.0 speed. Even my
compromise proposal doesn't have a volunteer at the moment...
Gotta dig back thru my mail... I forget who has been maintaining the proxy
module. Maybe they're up for moving all/some of it to 2.0.
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
Re: axe proxy support from 2.0?
Posted by Eli Marmor <ma...@elmar.co.il>.
David Reid wrote:
> I think the point that Gregs been making (and he'll correct me if I'm wrong
> I'm sure) is that what most people want is the functionality of various bits
> of a proxy. Given this it seems that if we concentrate on writing those
> bits to be tightly integrated with Apache-2.0 and abandon the idea that we
> supply a "full-blown" proxy we're getting more bang for our buck.
My message was not a response to Greg's message (BTW: I think that he's
doing some amazing projects, like the mod_dav and other useful things);
It was a response to the whole thread/idea, and his message was only the
last one I saw when I decided to write my message. In any case, I under-
stood the point that you are trying to clarify, when I wrote the
previous message, and still thought that many people would miss that
stuff. Again, I have good reasons for my opinion, but no time and/or
desire to argue about them. I think that the fact that many people rely
on things which are not among the "various bits" you mentioned, is
enough to leave the whole proxy module as is, and logical reasons are
not needed.
--
Eli Marmor
marmor@elmar.co.il
El-Mar Software Ltd.
Re: axe proxy support from 2.0?
Posted by David Reid <ab...@dial.pipex.com>.
I think the point that Gregs been making (and he'll correct me if I'm wrong
I'm sure) is that what most people want is the functionality of various bits
of a proxy. Given this it seems that if we concentrate on writing those
bits to be tightly integrated with Apache-2.0 and abandon the idea that we
supply a "full-blown" proxy we're getting more bang for our buck.
Just my 2p again...
david
----- Original Message -----
From: Eli Marmor <ma...@elmar.co.il>
To: <ne...@apache.org>
Sent: 13 November 1999 17:23
Subject: Re: axe proxy support from 2.0?
> Hello,
>
> I didn't express my opinion in this issue so far; I don't think that
> my opinion is so important, I'm not one of the core developers, my
> contributions to the development of Apache were too humble, and I
> didn't want to make anybody angry.
>
> But I see that this thread is going and going, and without expressing
> my opinion or why I think that it is very dangerous to axe the proxy
> stuff (I don't want and don't have time to), I just wanted to let you
> know that I, and many other people and companies I know, use this
> stuff very heavily. Squid can't help, because in most of the cases
> the purpose is to use the same program for some HTTP purposes AND
> proxing.
>
> I remember that less controversal proposals, such as integration of
> the EAPI code (please don't flame; Just an example!), were denied
> very easily, although most of the developers wanted them. In some
> cases, even an objection of ONE developer, sent an idea to the
> trashcan (these are the rules, aren't?).
>
> Of course, we can compromise on putting the proxy stuff in the
> "experimental" directory. But it is exactly as integrating EAPI into
> the code while excluding the "-DEAPI" from the default configuration
> (again, I used EAPI only as an example).
>
> I think that if it is so critical and important to axe it out, maybe
> it will be better to just add a disclaimer (like in the case of the
> WIN32 port) that it is not 100% stable (although IMHO it IS), or add
> a recommendation to use Squid. In any case, don't forget that the
> default configuration of httpd.conf is "ProxyRequests Off", and
> nobody is forced to uncomment the "#ProxyRequests On". So even a
> disclaimer and/or recommendation to use Squid, is already a huge
> compromise.
>
> --
> Eli Marmor
> marmor@elmar.co.il
> El-Mar Software Ltd.
Re: axe proxy support from 2.0?
Posted by Ben Laurie <be...@algroup.co.uk>.
Eli Marmor wrote:
> I remember that less controversal proposals, such as integration of
> the EAPI code (please don't flame; Just an example!), were denied
> very easily, although most of the developers wanted them. In some
> cases, even an objection of ONE developer, sent an idea to the
> trashcan (these are the rules, aren't?).
This is a bad example - there's equivalent functionality already in 2.0.
Failing to use a particular implementation isn't the same as failing to
provide particular functionality.
Cheers,
Ben.
--
http://www.apache-ssl.org/ben.html
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
- Indira Gandhi
Re: axe proxy support from 2.0?
Posted by Ryan Bloom <rb...@raleigh.ibm.com>.
Time for me to jump in here. I could be wrong, but I don't think
"ap_remote_retrieve" or any other functions specifically related to web
serving belong in APR.
APR is meant to be a general purpose Portable Run-time. It is not meant
to provide Apache with a list of functions for EVERYTHING. I think a well
thought-out ap_remote_retrieve would be fine, but it belongs in Apache.
I know the proxy module needs to be completely re-written if it is going
to survive, and it really should be updated to a 1.1 proxy, but pushing
functions that only make sense for a web server into APR is not the way to
accomplish this.
Ryan
On Sat, 13 Nov 1999, David Reid wrote:
> Maybe I'm well off the mark here, but if we add an "ap_remote_retrieve"
> function to APR and add the necessary support into Apache, couldn't a
> rewritten proxy module make use of it? If that's the case then I suggest we
> start out along the road of adding the remote fetch capability then once
> things are more stable we can re-assess whether or not a full blown proxy
> module is required. In the mean time why not create a subdirectory
> alongside experimental called "old" or some such and move the proxy stuff in
> there until we figure out in the fullness of time what's happening with it.
>
> How does that sound to everyone. We're all agreed that the remote fetch
> stuff will be useful, so I see no real problems bar someone volunteering to
> add the functions to APR. Don't really want to drop it onto Ryan, he's
> already got too much to do! No, I'm not volunteering either!
>
> Just another 2p worth...
>
> david
> ----- Original Message -----
> From: Greg Stein <gs...@lyra.org>
> To: <ne...@apache.org>
> Sent: 13 November 1999 19:54
> Subject: Re: axe proxy support from 2.0?
>
>
> > On Sat, 13 Nov 1999, Eli Marmor wrote:
> > > I didn't express my opinion in this issue so far; I don't think that
> > > my opinion is so important, I'm not one of the core developers, my
> > > contributions to the development of Apache were too humble, and I
> > > didn't want to make anybody angry.
> >
> > As I mentioned previously... your opinion (and others') is always
> > valuable. It provides some guide before we go and do something silly :-)
> >
> > >...
> > > I remember that less controversal proposals, such as integration of
> > > the EAPI code (please don't flame; Just an example!), were denied
> > > very easily, although most of the developers wanted them. In some
> > > cases, even an objection of ONE developer, sent an idea to the
> > > trashcan (these are the rules, aren't?).
> >
> > Yes, any developer can veto any change.
> >
> > One of the issues with EAPI was timing, rather than issues with the patch
> > itself. I believe it stands a much better chance for 2.0 :-)
> >
> > (note that hooks are in 2.0 already and MM is sitting there but not fully
> > folded in yet)
> >
> > >...
> > > I think that if it is so critical and important to axe it out, maybe
> > > it will be better to just add a disclaimer (like in the case of the
> > > WIN32 port) that it is not 100% stable (although IMHO it IS), or add
> > > a recommendation to use Squid. In any case, don't forget that the
> > > default configuration of httpd.conf is "ProxyRequests Off", and
> > > nobody is forced to uncomment the "#ProxyRequests On". So even a
> > > disclaimer and/or recommendation to use Squid, is already a huge
> > > compromise.
> >
> > At the moment, the disclaimer would be "this code might not even compile,
> > let alone function." While it may be stable for 1.3, we're talking 2.0
> > here.
> >
> > (actually, I don't have a real handle on how *close* the code is to
> > working; it might work fine for all I know; part of my impetus for change
> > is to reduce our bug count caused by something that I don't feel "belongs"
> > in a web server; the remote-fetching concept makes good sense, tho)
> >
> > So it isn't really so much "critical/important" to axe it, as it is a
> > recognition that something needs to be done.
> >
> > Cheers,
> > -g
> >
> > --
> > Greg Stein, http://www.lyra.org/
> >
> >
>
_______________________________________________________________________
Ryan Bloom rbb@raleigh.ibm.com
4205 S Miami Blvd
RTP, NC 27709 It's a beautiful sight to see good dancers
doing simple steps. It's a painful sight to
see beginners doing complicated patterns.
Re: axe proxy support from 2.0?
Posted by David Reid <ab...@dial.pipex.com>.
Maybe I'm well off the mark here, but if we add an "ap_remote_retrieve"
function to APR and add the necessary support into Apache, couldn't a
rewritten proxy module make use of it? If that's the case then I suggest we
start out along the road of adding the remote fetch capability then once
things are more stable we can re-assess whether or not a full blown proxy
module is required. In the mean time why not create a subdirectory
alongside experimental called "old" or some such and move the proxy stuff in
there until we figure out in the fullness of time what's happening with it.
How does that sound to everyone. We're all agreed that the remote fetch
stuff will be useful, so I see no real problems bar someone volunteering to
add the functions to APR. Don't really want to drop it onto Ryan, he's
already got too much to do! No, I'm not volunteering either!
Just another 2p worth...
david
----- Original Message -----
From: Greg Stein <gs...@lyra.org>
To: <ne...@apache.org>
Sent: 13 November 1999 19:54
Subject: Re: axe proxy support from 2.0?
> On Sat, 13 Nov 1999, Eli Marmor wrote:
> > I didn't express my opinion in this issue so far; I don't think that
> > my opinion is so important, I'm not one of the core developers, my
> > contributions to the development of Apache were too humble, and I
> > didn't want to make anybody angry.
>
> As I mentioned previously... your opinion (and others') is always
> valuable. It provides some guide before we go and do something silly :-)
>
> >...
> > I remember that less controversal proposals, such as integration of
> > the EAPI code (please don't flame; Just an example!), were denied
> > very easily, although most of the developers wanted them. In some
> > cases, even an objection of ONE developer, sent an idea to the
> > trashcan (these are the rules, aren't?).
>
> Yes, any developer can veto any change.
>
> One of the issues with EAPI was timing, rather than issues with the patch
> itself. I believe it stands a much better chance for 2.0 :-)
>
> (note that hooks are in 2.0 already and MM is sitting there but not fully
> folded in yet)
>
> >...
> > I think that if it is so critical and important to axe it out, maybe
> > it will be better to just add a disclaimer (like in the case of the
> > WIN32 port) that it is not 100% stable (although IMHO it IS), or add
> > a recommendation to use Squid. In any case, don't forget that the
> > default configuration of httpd.conf is "ProxyRequests Off", and
> > nobody is forced to uncomment the "#ProxyRequests On". So even a
> > disclaimer and/or recommendation to use Squid, is already a huge
> > compromise.
>
> At the moment, the disclaimer would be "this code might not even compile,
> let alone function." While it may be stable for 1.3, we're talking 2.0
> here.
>
> (actually, I don't have a real handle on how *close* the code is to
> working; it might work fine for all I know; part of my impetus for change
> is to reduce our bug count caused by something that I don't feel "belongs"
> in a web server; the remote-fetching concept makes good sense, tho)
>
> So it isn't really so much "critical/important" to axe it, as it is a
> recognition that something needs to be done.
>
> Cheers,
> -g
>
> --
> Greg Stein, http://www.lyra.org/
>
>
Re: axe proxy support from 2.0?
Posted by Greg Stein <gs...@lyra.org>.
On Sat, 13 Nov 1999, Eli Marmor wrote:
> I didn't express my opinion in this issue so far; I don't think that
> my opinion is so important, I'm not one of the core developers, my
> contributions to the development of Apache were too humble, and I
> didn't want to make anybody angry.
As I mentioned previously... your opinion (and others') is always
valuable. It provides some guide before we go and do something silly :-)
>...
> I remember that less controversal proposals, such as integration of
> the EAPI code (please don't flame; Just an example!), were denied
> very easily, although most of the developers wanted them. In some
> cases, even an objection of ONE developer, sent an idea to the
> trashcan (these are the rules, aren't?).
Yes, any developer can veto any change.
One of the issues with EAPI was timing, rather than issues with the patch
itself. I believe it stands a much better chance for 2.0 :-)
(note that hooks are in 2.0 already and MM is sitting there but not fully
folded in yet)
>...
> I think that if it is so critical and important to axe it out, maybe
> it will be better to just add a disclaimer (like in the case of the
> WIN32 port) that it is not 100% stable (although IMHO it IS), or add
> a recommendation to use Squid. In any case, don't forget that the
> default configuration of httpd.conf is "ProxyRequests Off", and
> nobody is forced to uncomment the "#ProxyRequests On". So even a
> disclaimer and/or recommendation to use Squid, is already a huge
> compromise.
At the moment, the disclaimer would be "this code might not even compile,
let alone function." While it may be stable for 1.3, we're talking 2.0
here.
(actually, I don't have a real handle on how *close* the code is to
working; it might work fine for all I know; part of my impetus for change
is to reduce our bug count caused by something that I don't feel "belongs"
in a web server; the remote-fetching concept makes good sense, tho)
So it isn't really so much "critical/important" to axe it, as it is a
recognition that something needs to be done.
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
Re: axe proxy support from 2.0?
Posted by Eli Marmor <ma...@elmar.co.il>.
Hello,
I didn't express my opinion in this issue so far; I don't think that
my opinion is so important, I'm not one of the core developers, my
contributions to the development of Apache were too humble, and I
didn't want to make anybody angry.
But I see that this thread is going and going, and without expressing
my opinion or why I think that it is very dangerous to axe the proxy
stuff (I don't want and don't have time to), I just wanted to let you
know that I, and many other people and companies I know, use this
stuff very heavily. Squid can't help, because in most of the cases
the purpose is to use the same program for some HTTP purposes AND
proxing.
I remember that less controversal proposals, such as integration of
the EAPI code (please don't flame; Just an example!), were denied
very easily, although most of the developers wanted them. In some
cases, even an objection of ONE developer, sent an idea to the
trashcan (these are the rules, aren't?).
Of course, we can compromise on putting the proxy stuff in the
"experimental" directory. But it is exactly as integrating EAPI into
the code while excluding the "-DEAPI" from the default configuration
(again, I used EAPI only as an example).
I think that if it is so critical and important to axe it out, maybe
it will be better to just add a disclaimer (like in the case of the
WIN32 port) that it is not 100% stable (although IMHO it IS), or add
a recommendation to use Squid. In any case, don't forget that the
default configuration of httpd.conf is "ProxyRequests Off", and
nobody is forced to uncomment the "#ProxyRequests On". So even a
disclaimer and/or recommendation to use Squid, is already a huge
compromise.
--
Eli Marmor
marmor@elmar.co.il
El-Mar Software Ltd.
Re: axe proxy support from 2.0?
Posted by Greg Stein <gs...@lyra.org>.
On 12 Nov 1999 tvaughan@aventail.com wrote:
> Greg Stein <gs...@lyra.org> writes:
> > Now, to be clear: while I'd be happy to be the guy to rip proxy support
> > (fair enough that if I start a discussion to rip it out, then I better be
> > ready to volunteer for it :-), I'm not standing up to do the work for this
> > new stuff. Certainly, if somebody wants to just port over mod_proxy and do
> > the whole biz, then they're welcome to. I'm simply trying to shepherd the
> > requirements around bringing the proxy stuff into 2.0.
>
> I think it would be helpful to say which of these are more appropriate for
> apr too.
I was deferring that until after some semblance of consensus arrived :-)
But hey... all right.
> Like how about an http client library? One shared by ab and this
> new "shiny object". Or perhaps this is a new library built on top of
> apr. And hooks to filter fetched resources would be nice too. Not that I
> can promise any of this myself either, but should I happen to have a spare
> weekend, it would be nice to know what would be accepted.
In one of my prior emails, I was going to say "just grab one of the HTTP
client libraries that are out there" but then realized that we'd probably
want to use APR.
I would think that a library built on APR would be most appropriate.
Filtering or not is pretty easy. To filter:
while (1) {
read data
filter
send data
}
To just copy a remote resource:
ap_sendfile(...)
In other words, the HTTP client lib probably ought to just expose an
ap_socket_t for the body rather than a higher-level concept.
(note that it might do some processing on the headers, though!)
> > Also note that I advocate removing forward proxy support. Many of the bugs
> > we have seem to deal with protocol issues around this. I would also
> > advocate killing FTP support for remote resources.
>
> /I/ can live with this.
Excellent!
I'd love to be in a world with no more FTP. Just give me DAV...
On Fri, 12 Nov 1999, Jim Winstead wrote:
> On Nov 12, Greg Stein wrote:
> > I'm currently advocating dropping the term/thought/belief that we're a
> > forward/reverse proxy and just support fetching remote resources (with
> > optional/smart caching of those resources).
>
> And where 'remote' can simply mean another port on the local machine.
> A bunch of mod_perl-enabled servers fronted by lightweight static/proxy
> servers, for example.
Sure. That would seem to be straight-forward, as I don't think the remote
fetching would care whether the target host is remote/local.
> In that case, it can also be useful to be able to pass some additional
> things to and from the backend server (so they can be logged by
> the front-end, for example). We've got some hacked-up solutions to
> that, but it would be nice to see a clean mainstream solution.
I think this would be a pretty straight-forward extension. Given an HTTP
client lib (as above), then the module that *uses* that lib could quite
easily expose various Apache config commands to insert stuff into the
request stream and/or parse headers/body from the response.
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
Re: axe proxy support from 2.0?
Posted by Jim Winstead <ji...@trainedmonkey.com>.
On Nov 12, Greg Stein wrote:
> I'm currently advocating dropping the term/thought/belief that we're a
> forward/reverse proxy and just support fetching remote resources (with
> optional/smart caching of those resources).
And where 'remote' can simply mean another port on the local machine.
A bunch of mod_perl-enabled servers fronted by lightweight static/proxy
servers, for example.
In that case, it can also be useful to be able to pass some additional
things to and from the backend server (so they can be logged by
the front-end, for example). We've got some hacked-up solutions to
that, but it would be nice to see a clean mainstream solution.
Jim
Re: axe proxy support from 2.0?
Posted by Greg Stein <gs...@lyra.org>.
On Fri, 12 Nov 1999, Michael Hall wrote:
> > Greg Stein wrote:
> > > This came up a while back, and there is actually an item in the STATUS
> > > file, apparently (Manoj said so :-), but why do we still have the proxy
> > > support in Apache? It seems like it has a largish number of bugs and is
> > > pretty orthogonal to the rest of Apache. Can't we just axe it and refer
> > > people to Squid?
>...
> Anyway, hadn't planned on jumping in yet being new but as an Apache,
> mod_perl user I also have to say I'd like to see some incarnation of mod_
> proxy. A lot of us use a two server setup with a standard light-weight
> Apache setup using mod-rewrite (or ProxyPass) and ReverseProxyPass as a
> front-end for the heavy-weight Apache/mod_perl backend.
I believe that my modified proposal would still work for you.
I'm currently advocating dropping the term/thought/belief that we're a
forward/reverse proxy and just support fetching remote resources (with
optional/smart caching of those resources).
In other words, rather than using Proxy directives (which is really a
misnomer for the way it seems most people use this stuff), you might
(instead) have something like this:
RemoteAlias /my-other-server http://my-other-server/public
Caching could be determined something like this:
<Location /my-other-server>
CacheSize 5M
CacheDuration 3h
</Location>
(making up parameters here...)
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
Re: axe proxy support from 2.0?
Posted by Michael Hall <mh...@riverside.org>.
On Fri, Nov 12, 1999 at 09:29:30AM +0100, Graham Leggett wrote:
> Greg Stein wrote:
>
> > This came up a while back, and there is actually an item in the STATUS
> > file, apparently (Manoj said so :-), but why do we still have the proxy
> > support in Apache? It seems like it has a largish number of bugs and is
> > pretty orthogonal to the rest of Apache. Can't we just axe it and refer
> > people to Squid?
> >
> > It seems like there are sufficient alternatives in the open-source world
> > that we shouldn't try to pretend to be proxy developers...
>
> Aaaargh - no! Don't you dare... (much pleading, etc).
Just a quick introduction, I've been lurking on the list, compiling and
playing with the 2.0 CVS snapshots. I'm not a programmer so I can't contribute
anything in that respect (fortunately if you saw how I mangle C :-) but as
a long time Apache and mod-perl user I'm naturally interested in following
new developments.
Anyway, hadn't planned on jumping in yet being new but as an Apache,
mod_perl user I also have to say I'd like to see some incarnation of mod_
proxy. A lot of us use a two server setup with a standard light-weight
Apache setup using mod-rewrite (or ProxyPass) and ReverseProxyPass as a
front-end for the heavy-weight Apache/mod_perl backend.
--
If Limbaugh is the answer, it must be a stupid question.
Mike Hall <mh...@riverside.org>, ICQ: #37292579, http://www.riverside.org
System Administrator (MH993) (*nix, OS/2 certified - C, Perl, CGI hacker)
Re: axe proxy support from 2.0?
Posted by tv...@aventail.com.
Greg Stein <gs...@lyra.org> writes:
> Now, to be clear: while I'd be happy to be the guy to rip proxy support
> (fair enough that if I start a discussion to rip it out, then I better be
> ready to volunteer for it :-), I'm not standing up to do the work for this
> new stuff. Certainly, if somebody wants to just port over mod_proxy and do
> the whole biz, then they're welcome to. I'm simply trying to shepherd the
> requirements around bringing the proxy stuff into 2.0.
I think it would be helpful to say which of these are more appropriate for
apr too. Like how about an http client library? One shared by ab and this
new "shiny object". Or perhaps this is a new library built on top of
apr. And hooks to filter fetched resources would be nice too. Not that I
can promise any of this myself either, but should I happen to have a spare
weekend, it would be nice to know what would be accepted.
>
> Also note that I advocate removing forward proxy support. Many of the bugs
> we have seem to deal with protocol issues around this. I would also
> advocate killing FTP support for remote resources.
/I/ can live with this.
-Tom
Re: axe proxy support from 2.0?
Posted by David Reid <ab...@dial.pipex.com>.
As cahing of local items has already been raised a few times this seems to
be a logical inclusion.
d.
----- Original Message -----
From: Greg Stein <gs...@lyra.org>
To: <ne...@apache.org>
Sent: 12 November 1999 22:38
Subject: Re: axe proxy support from 2.0?
> Absolutely. That is what I'm trying to boil it down to. First step was
> "nuke the proxy stuff [because nobody wants it]". Of course, then people
> speak up saying they DO want it and describe why. Second step is "let's
> allow fetching of remote resources [but not call, try, or pretend we're a
> proxy]". This seems to fit much better with people. Third step is "make
> sure we have a cache in there."
>
> In other words, here is where I think we are for 2.0 w.r.t. the old proxy
> module:
>
> * axe "proxy" support (forward/reverse)
> * port over (write from scratch) fetching of remote resources. possibly
> simplified in that we won't act as a true proxy, but an HTTP (FTP?)
> client library.
> [Roy: must we act as a proxy, or can Apache pretend it is the origin
> server and keep the client unawares?]
> * port over (write from scratch) caching of fetched resources. possibly
> allow caching of other responses and local sources.
>
> That's it.
>
> Now, to be clear: while I'd be happy to be the guy to rip proxy support
> (fair enough that if I start a discussion to rip it out, then I better be
> ready to volunteer for it :-), I'm not standing up to do the work for this
> new stuff. Certainly, if somebody wants to just port over mod_proxy and do
> the whole biz, then they're welcome to. I'm simply trying to shepherd the
> requirements around bringing the proxy stuff into 2.0.
>
> Also note that I advocate removing forward proxy support. Many of the bugs
> we have seem to deal with protocol issues around this. I would also
> advocate killing FTP support for remote resources.
>
> Cheers,
> -g
>
> On Fri, 12 Nov 1999, David Reid wrote:
> > Rather than discussing NOT removing something, should we talk about what
we
> > WANT in the new "proxy/file getter/cache/whatever we want to call it"?
> >
> > Might be more productive and then sets the targets for someone to
start...
> >
> > Just my 2p...
> >
> > david
> > ----- Original Message -----
> > From: <tv...@aventail.com>
> > To: <ne...@apache.org>
> > Sent: 12 November 1999 18:17
> > Subject: Re: axe proxy support from 2.0?
> >
> >
> > > Graham Leggett <mi...@sharp.fm> writes:
> > >
> > > > Greg Stein wrote:
> > > >
> > > > > All right. Axe the proxy thing. Build in specific support for
grabbing
> > > > > files from elsewhere. If that goes best into the layering, as
another
> > APR
> > > > > type, or whatever, then it is clear what is going on.
> > > > >
> > > > > It seems that we could simplify a lot of our life if we stopped
> > viewing it
> > > > > as a (reverse) proxy and viewed/built it as a set of
> > > > > functions/types/whatever for fetching remote files. Certainly a
lot
> > easier
> > > > > to build (rather than porting/rebuilding mod_proxy for 2.0).
> > > >
> > > > This is true - though this encompasses about 90% of the existing
proxy
> > > > code anyway. Remember that the ftp and http "client" bits as well as
the
> > > > caching engine all have significant use inside the reverse proxy,
> > > > especially the caching - it makes a huge difference on a busy
server.
> > > > Even the parent cache support is useful to connect the reverse proxy
> > > > through a further proxy.
> > >
> > > The cache is key. We've seen latency times increase by as much as a
factor
> > > of six without it. (The more cache friendly an origin server is, the
> > > better.) It might be useful to abstract the cache out as well. I
suppose
> > > mod_include might want to cache a document it somehow knows can be
cached.
> > >
> > > The ability to add other protocols, like ones that start with 'H' and
end
> > > with 'S', would be nice too.
> > >
> > > -Tom
> > >
> > > --
> > > Tom Vaughan <tvaughan at aventail dot com>
> >
>
> --
> Greg Stein, http://www.lyra.org/
>
Re: axe proxy support from 2.0?
Posted by Greg Stein <gs...@lyra.org>.
Absolutely. That is what I'm trying to boil it down to. First step was
"nuke the proxy stuff [because nobody wants it]". Of course, then people
speak up saying they DO want it and describe why. Second step is "let's
allow fetching of remote resources [but not call, try, or pretend we're a
proxy]". This seems to fit much better with people. Third step is "make
sure we have a cache in there."
In other words, here is where I think we are for 2.0 w.r.t. the old proxy
module:
* axe "proxy" support (forward/reverse)
* port over (write from scratch) fetching of remote resources. possibly
simplified in that we won't act as a true proxy, but an HTTP (FTP?)
client library.
[Roy: must we act as a proxy, or can Apache pretend it is the origin
server and keep the client unawares?]
* port over (write from scratch) caching of fetched resources. possibly
allow caching of other responses and local sources.
That's it.
Now, to be clear: while I'd be happy to be the guy to rip proxy support
(fair enough that if I start a discussion to rip it out, then I better be
ready to volunteer for it :-), I'm not standing up to do the work for this
new stuff. Certainly, if somebody wants to just port over mod_proxy and do
the whole biz, then they're welcome to. I'm simply trying to shepherd the
requirements around bringing the proxy stuff into 2.0.
Also note that I advocate removing forward proxy support. Many of the bugs
we have seem to deal with protocol issues around this. I would also
advocate killing FTP support for remote resources.
Cheers,
-g
On Fri, 12 Nov 1999, David Reid wrote:
> Rather than discussing NOT removing something, should we talk about what we
> WANT in the new "proxy/file getter/cache/whatever we want to call it"?
>
> Might be more productive and then sets the targets for someone to start...
>
> Just my 2p...
>
> david
> ----- Original Message -----
> From: <tv...@aventail.com>
> To: <ne...@apache.org>
> Sent: 12 November 1999 18:17
> Subject: Re: axe proxy support from 2.0?
>
>
> > Graham Leggett <mi...@sharp.fm> writes:
> >
> > > Greg Stein wrote:
> > >
> > > > All right. Axe the proxy thing. Build in specific support for grabbing
> > > > files from elsewhere. If that goes best into the layering, as another
> APR
> > > > type, or whatever, then it is clear what is going on.
> > > >
> > > > It seems that we could simplify a lot of our life if we stopped
> viewing it
> > > > as a (reverse) proxy and viewed/built it as a set of
> > > > functions/types/whatever for fetching remote files. Certainly a lot
> easier
> > > > to build (rather than porting/rebuilding mod_proxy for 2.0).
> > >
> > > This is true - though this encompasses about 90% of the existing proxy
> > > code anyway. Remember that the ftp and http "client" bits as well as the
> > > caching engine all have significant use inside the reverse proxy,
> > > especially the caching - it makes a huge difference on a busy server.
> > > Even the parent cache support is useful to connect the reverse proxy
> > > through a further proxy.
> >
> > The cache is key. We've seen latency times increase by as much as a factor
> > of six without it. (The more cache friendly an origin server is, the
> > better.) It might be useful to abstract the cache out as well. I suppose
> > mod_include might want to cache a document it somehow knows can be cached.
> >
> > The ability to add other protocols, like ones that start with 'H' and end
> > with 'S', would be nice too.
> >
> > -Tom
> >
> > --
> > Tom Vaughan <tvaughan at aventail dot com>
>
--
Greg Stein, http://www.lyra.org/
Re: OFF TOPIC: Any general apache administration mail lists?
Posted by Ask Bjoern Hansen <as...@valueclick.com>.
On Sun, 14 Nov 1999, Manoj Kasichainula wrote:
[...]
> If this is important to you, use a piped logger which sends all
> entries to a central machine for writing.
at ValueClick we use a system like that (s/piped logger/custom apache
module/) for all our logging. Works very well and it was easy to do all
sorts of buffering and stuff for reliability. And in the normal case there
is no disk IO from logging on the webserver (and no multi gigabyte files
to rotate and stuff).
- ask
--
ask bjoern hansen - <http://www.netcetera.dk/~ask/>
more than 60M impressions per day, <http://valueclick.com>
Re: OFF TOPIC: Any general apache administration mail lists?
Posted by Manoj Kasichainula <ma...@io.com>.
On Sun, Nov 14, 1999 at 10:16:00AM -0500, Jeremy Hansen wrote:
> I understand what you're saying. Could you explain reason why the logs
> writes are *not* locked?
Because they don't have to be. Writes of a certain length are
guaranteed to be atomic. I can't speak for the enjoyable experience
that is NFS, though.
It's not worth the slowdown to lock logs before writing a log entry.
And I believe Apache would have to close and reopen the log file for
every sinle entry for this to work, which is completely insane.
As many people have said already, use separate logs on each server and
combine them before doing log analysis.
> Now I need to write ugly scripts to combine log files
cat log1 log2 log3 > newlog
Log files are also not guaranteed to be ordered, so if your log
analyzer requires that, you'd have to write a script to sort the
entries anyway.
> and it pretty much throws a lot of the
> functionality of WebTrends' auto scheduling and updating out the window
> unless I have my logs sync every 15 minutes.
If this is important to you, use a piped logger which sends all
entries to a central machine for writing.
--
Manoj Kasichainula - manojk at io dot com - http://www.io.com/~manojk/
Re: OFF TOPIC: Any general apache administration mail lists?
Posted by Jeremy Hansen <je...@xxedgexx.com>.
Thanks for the responses. I understand better now. Never really
considered the performance cost of locking log writes from Apache's
perspective.
I think I've come up with a solution though that would allow me to let
things be on separate machines and not have a cheesy script to move things
around.
Let's say I have each web server write to /var/log/httpd which is not
shared via nfs, then on the main server were web trends is running I could
have it nfs mount each web server:
webtrend:/var/log/httpd/{web1,web2,web3}
This would allow me to just use multiple log files in webtrends for one
profile, which you can do simple by separating the files by pipes in the
profile.
Hmm, that would work well, allow all writes to be local and not have the
need of writing scripts to gather and combine and this way I can still use
webtrends' scheduling without fear of things being out of sync. Cool.
Well, thanks for the help. Sorry it took me so long to "get it", but I
perceived things to be much easier then what they really are.
-jeremy
> At 10:16am Nov 14, 1999, Jeremy Hansen wrote:
>
> > I understand what you're saying. Could you explain reason why the
> > logs writes are *not* locked?
>
> Performance. If Apache has to be so careful about different pools stepping
> on each others' toes, that's wasted effort. Reliability. What about pool
> failures, stuck NFS connections, etc.? Would there be a possibility that a
> misbehaving pool could lock the log file and in doing so cripple the pools
> running on other servers? Would using a common logfile reintroduce the
> sort of single point of failure that multiple-server setups are supposed
> to prevent?
>
> > Now I need to write ugly scripts to combine log files and it pretty
> > much throws a lot of the functionality of WebTrends' auto scheduling
> > and updating out the window unless I have my logs sync every 15
> > minutes. It just looks ugly to me, that's all.
>
> Talk to the folks at WebTrends. It is very, very typical for busy Web
> sites to have multiple servers, each recording their own set of log files.
> If WebTrends can't handle that, then thay should see an "opportunity"
> here. ;-)
>
> -Peter
>
http://www.xxedgexx.com | jeremy@xxedgexx.com
---------------------------------------------
Y2K. We're all gonna die.
Re: OFF TOPIC: Any general apache administration mail lists?
Posted by Peter W <pe...@usa.net>.
At 10:16am Nov 14, 1999, Jeremy Hansen wrote:
> I understand what you're saying. Could you explain reason why the
> logs writes are *not* locked?
Performance. If Apache has to be so careful about different pools stepping
on each others' toes, that's wasted effort. Reliability. What about pool
failures, stuck NFS connections, etc.? Would there be a possibility that a
misbehaving pool could lock the log file and in doing so cripple the pools
running on other servers? Would using a common logfile reintroduce the
sort of single point of failure that multiple-server setups are supposed
to prevent?
> Now I need to write ugly scripts to combine log files and it pretty
> much throws a lot of the functionality of WebTrends' auto scheduling
> and updating out the window unless I have my logs sync every 15
> minutes. It just looks ugly to me, that's all.
Talk to the folks at WebTrends. It is very, very typical for busy Web
sites to have multiple servers, each recording their own set of log files.
If WebTrends can't handle that, then thay should see an "opportunity"
here. ;-)
-Peter
Re: OFF TOPIC: Any general apache administration mail lists?
Posted by Dean Gaudet <dg...@arctic.org>.
On Sun, 14 Nov 1999, Jeremy Hansen wrote:
> [why isn't NFS like a unix filesystem?]
people have been asking that forever. it just isn't. expecting NFS to
behave like local disk will get you into lots of troubles. say no.
Dean
Re: OFF TOPIC: Any general apache administration mail lists?
Posted by Jeremy Hansen <je...@xxedgexx.com>.
I understand what you're saying. Could you explain reason why the logs
writes are *not* locked?
Also a few people seem to have done what I'm trying to do although it
sound like it may be other platforms besides Linux. When I attempt to put
the lock file on an nfs volume, any ideas why httpd goes into D state and
the serve hangs?
I'm going to just write to local disks now, but this just seems like
something that should be possible. I don't know if it's apache's fault or
Linux NFS fault, I'm not a developer but all I keep hearing is "you don't
get it, it's not supposed to do that" which is fine, but it seems like
functionality that could be taken advantage of. Now I need to write ugly
scripts to combine log files and it pretty much throws a lot of the
functionality of WebTrends' auto scheduling and updating out the window
unless I have my logs sync every 15 minutes. It just looks ugly to me,
that's all.
Thanks
-jeremy
> Jeremy Hansen wrote:
> >
> > Again, if this is what I need to do, then I guess I will, but I'dd like to
> > hear more feedback. Also I'm confused as to what you say about LockFile.
> > This is what the docs read:
> >
> > The main reason for changing it is if the logs directory is NFS mounted,
> > since the lockfile must be stored on a local disk. The PID of the main
> > server process is automatically appended to the filename.
>
> I think you are completely missing the point here ... the lock file is
> used (when it is used at all) to serialise the accept() function, which
> some OSes don't like having multiple processes use at the same time on
> the same socket. It has _absolutely nothing to do with logs_, except
> that by default it goes in the same directory.
>
> Logfiles are simply not locked. They're written with a single write(),
> which is assumed to be atomic. I have no doubt that under NFS it ain't.
>
> > This sound like what I'm doing, is it not? Also like I said, using
> > USE_FCNTL_SERIALIZED_ACCEPT I see no httpd.lock file created, I only see
> > it with USE_FLOCK_SERIALIZED_ACCEPT.
>
> ISTR httpd.lock is unlinked with fcntl(), but can't be with flock(). But
> I could be wrong.
>
> Cheers,
>
> Ben.
>
> --
> http://www.apache-ssl.org/ben.html
>
> "My grandfather once told me that there are two kinds of people: those
> who work and those who take the credit. He told me to try to be in the
> first group; there was less competition there."
> - Indira Gandhi
>
http://www.xxedgexx.com | jeremy@xxedgexx.com
---------------------------------------------
Y2K. We're all gonna die.
Re: OFF TOPIC: Any general apache administration mail lists?
Posted by Ben Laurie <be...@algroup.co.uk>.
Jeremy Hansen wrote:
>
> Again, if this is what I need to do, then I guess I will, but I'dd like to
> hear more feedback. Also I'm confused as to what you say about LockFile.
> This is what the docs read:
>
> The main reason for changing it is if the logs directory is NFS mounted,
> since the lockfile must be stored on a local disk. The PID of the main
> server process is automatically appended to the filename.
I think you are completely missing the point here ... the lock file is
used (when it is used at all) to serialise the accept() function, which
some OSes don't like having multiple processes use at the same time on
the same socket. It has _absolutely nothing to do with logs_, except
that by default it goes in the same directory.
Logfiles are simply not locked. They're written with a single write(),
which is assumed to be atomic. I have no doubt that under NFS it ain't.
> This sound like what I'm doing, is it not? Also like I said, using
> USE_FCNTL_SERIALIZED_ACCEPT I see no httpd.lock file created, I only see
> it with USE_FLOCK_SERIALIZED_ACCEPT.
ISTR httpd.lock is unlinked with fcntl(), but can't be with flock(). But
I could be wrong.
Cheers,
Ben.
--
http://www.apache-ssl.org/ben.html
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
- Indira Gandhi
Re: OFF TOPIC: Any general apache administration mail lists?
Posted by Greg Stein <gs...@lyra.org>.
Normally, Apache places the lock file into the logs directory. If the logs
directory is on an NFS mount (note that Apache can't put its lock file on
an NFS mount), then you must use LockFile to put the lock file on a local
disk.
The lock file is used for handling the accept() call on a web request. It
is unrelated to log files.
[ the fact that the lock file is normally placed into the log directory is
probably part of the confusion... ]
Now... how to get Apache to lock stuff properly on an NFS mount? Beats me.
I'm a neophyte in that whole area :-)
Cheers,
-g
On Fri, 12 Nov 1999, Jeremy Hansen wrote:
> Again, if this is what I need to do, then I guess I will, but I'dd like to
> hear more feedback. Also I'm confused as to what you say about LockFile.
> This is what the docs read:
>
> The main reason for changing it is if the logs directory is NFS mounted,
> since the lockfile must be stored on a local disk. The PID of the main
> server process is automatically appended to the filename.
>
> This sound like what I'm doing, is it not? Also like I said, using
> USE_FCNTL_SERIALIZED_ACCEPT I see no httpd.lock file created, I only see
> it with USE_FLOCK_SERIALIZED_ACCEPT.
>
> Thanks
> -jeremy
>
> > On Fri, 12 Nov 1999, Jeremy Hansen wrote:
> > > My LockFile currently points to a lock disk, /var/run, the default,
> > > although I never actually see a lock file being created. I switched to
> > > USE_FLOCK_blah_blah and then I did see the lock file being created, but
> > > Trond explained to me the flock is a BSD style lock and will not work with
> > > Linux NFS locking, only for local locks, and well it didn't work so...
> >
> > Well, the LockFile directive has nothing to do with locking on log file
> > writes. The LockFile specifies where the accept queue lock file should
> > go. This has to do with how incoming requests are handled.
> >
> > I don't know anything about nfs-based logs. I have never put myself in a
> > situation where I need to write to the same log file over nfs from many
> > different machines in rapid succession. The obvious thing I'd say is,
> > "Don't do that!" Most log file analyzers can merge log files at analysis
> > time. Given that, why not just make each server log to its own separate
> > file and avoid the locking issue?
> >
> > -Rasmus
> >
>
>
> http://www.xxedgexx.com | jeremy@xxedgexx.com
> ---------------------------------------------
> Y2K. We're all gonna die.
>
--
Greg Stein, http://www.lyra.org/
Re: OFF TOPIC: Any general apache administration mail lists?
Posted by Jeremy Hansen <je...@xxedgexx.com>.
Again, if this is what I need to do, then I guess I will, but I'dd like to
hear more feedback. Also I'm confused as to what you say about LockFile.
This is what the docs read:
The main reason for changing it is if the logs directory is NFS mounted,
since the lockfile must be stored on a local disk. The PID of the main
server process is automatically appended to the filename.
This sound like what I'm doing, is it not? Also like I said, using
USE_FCNTL_SERIALIZED_ACCEPT I see no httpd.lock file created, I only see
it with USE_FLOCK_SERIALIZED_ACCEPT.
Thanks
-jeremy
> On Fri, 12 Nov 1999, Jeremy Hansen wrote:
> > My LockFile currently points to a lock disk, /var/run, the default,
> > although I never actually see a lock file being created. I switched to
> > USE_FLOCK_blah_blah and then I did see the lock file being created, but
> > Trond explained to me the flock is a BSD style lock and will not work with
> > Linux NFS locking, only for local locks, and well it didn't work so...
>
> Well, the LockFile directive has nothing to do with locking on log file
> writes. The LockFile specifies where the accept queue lock file should
> go. This has to do with how incoming requests are handled.
>
> I don't know anything about nfs-based logs. I have never put myself in a
> situation where I need to write to the same log file over nfs from many
> different machines in rapid succession. The obvious thing I'd say is,
> "Don't do that!" Most log file analyzers can merge log files at analysis
> time. Given that, why not just make each server log to its own separate
> file and avoid the locking issue?
>
> -Rasmus
>
http://www.xxedgexx.com | jeremy@xxedgexx.com
---------------------------------------------
Y2K. We're all gonna die.
Re: OFF TOPIC: Any general apache administration mail lists?
Posted by Rasmus Lerdorf <ra...@apache.org>.
On Fri, 12 Nov 1999, Jeremy Hansen wrote:
> My LockFile currently points to a lock disk, /var/run, the default,
> although I never actually see a lock file being created. I switched to
> USE_FLOCK_blah_blah and then I did see the lock file being created, but
> Trond explained to me the flock is a BSD style lock and will not work with
> Linux NFS locking, only for local locks, and well it didn't work so...
Well, the LockFile directive has nothing to do with locking on log file
writes. The LockFile specifies where the accept queue lock file should
go. This has to do with how incoming requests are handled.
I don't know anything about nfs-based logs. I have never put myself in a
situation where I need to write to the same log file over nfs from many
different machines in rapid succession. The obvious thing I'd say is,
"Don't do that!" Most log file analyzers can merge log files at analysis
time. Given that, why not just make each server log to its own separate
file and avoid the locking issue?
-Rasmus
Re: OFF TOPIC: Any general apache administration mail lists?
Posted by Jim Winstead <ji...@trainedmonkey.com>.
Apache processes running on different servers shouldn't be using
the same accept lock file, so you gain nothing by putting it on an
NFS filesystem. That lockfile has nothing whatsoever to do with
any other file access the servers may do.
But this info from the open(2) man page might be enlightening:
O_APPEND
The file is opened in append mode. Initially, and
before each write, the file pointer is positioned
at the end of the file, as if with lseek.
O_APPEND may lead to corrupted files on NFS file
systems if more than one process appends data to a
file at once. This is because NFS does not sup-
port appending to a file, so the client kernel has
to simulate it, which can't be done without a race
condition.
Log to different files. Locking a log file just so you can write
to it would be a complete waste of time.
Jim
On Nov 12, Jeremy Hansen wrote:
>
> I know I can do this but my feeling is that I shouldn't have to and it
> adds a level of complexity to something that should work, shouldn't it?
>
> knfsd is supposed to support locking and it appears that it does, but from
> what I understand, Apache has to take advantage of the functions to do so.
>
> I don't know if I mentioned this, but Trond suggested that I ignore what
> Apache says and put the LockFile ON the nfs volume because having it on
> local disk was a 2.0.x kernel thing and doesn't apply in 2.2.x and knfs's
> case.
>
> I did that and all my httpd processes went into D state and the web
> servers all hung.
>
> If I have to, I guess writing to separate logs will be necessary, but it's
> my feeling that if I'm forced to do that, then there's a bug somewhere.
>
> -jeremy
>
> >
> > I can understand using nfs to share Web content on identical servers, esp.
> > if you have a really fast nfs toaster, but why don't you write separate
> > logs and recombine them before running WebTrends. IIRC, it tolerates
> > out-of-sequence log entries. If not, you could use a CustomLog format that
> > adds a timestamp at the beginning of the line, and then do a
> > merge->sort->trim routine at the end.
> >
> > -Peter
> >
> > At 3:43pm Nov 12, 1999, Jeremy Hansen wrote:
> >
> > > I have three web servers being load balanced. All the servers are
> > > writting their access logs over nfs to the same file. I'm using Linux
> > > knfsd with kernel's 2.2.12 from Red Hat 6.1 and kernel's 2.2.11-ac3 on the
> > > web servers with nfsv3 client patches from Trond applied.
> > >
> > > Basically I'm getting very corrupt log files. Half lines, lines running
> > > into each other, basically everything is telling me that locking is NOT
> > > working.
> >
> > > So...I don't know what else to explain. Like I said, if there's a better
> > > list, please let me know, but I'm going crazy trying to get this going. I
> > > have a perl script I wrote to weed out bad entries so we can run our
> > > reports, btw WebTrends sucks for its tolerance on log file errors, but I
> > > don't know if I'm loosing information or what.
> >
> >
>
>
> http://www.xxedgexx.com | jeremy@xxedgexx.com
> ---------------------------------------------
> Y2K. We're all gonna die.
>
>
Re: OFF TOPIC: Any general apache administration mail lists?
Posted by Jeremy Hansen <je...@xxedgexx.com>.
I know I can do this but my feeling is that I shouldn't have to and it
adds a level of complexity to something that should work, shouldn't it?
knfsd is supposed to support locking and it appears that it does, but from
what I understand, Apache has to take advantage of the functions to do so.
I don't know if I mentioned this, but Trond suggested that I ignore what
Apache says and put the LockFile ON the nfs volume because having it on
local disk was a 2.0.x kernel thing and doesn't apply in 2.2.x and knfs's
case.
I did that and all my httpd processes went into D state and the web
servers all hung.
If I have to, I guess writing to separate logs will be necessary, but it's
my feeling that if I'm forced to do that, then there's a bug somewhere.
-jeremy
>
> I can understand using nfs to share Web content on identical servers, esp.
> if you have a really fast nfs toaster, but why don't you write separate
> logs and recombine them before running WebTrends. IIRC, it tolerates
> out-of-sequence log entries. If not, you could use a CustomLog format that
> adds a timestamp at the beginning of the line, and then do a
> merge->sort->trim routine at the end.
>
> -Peter
>
> At 3:43pm Nov 12, 1999, Jeremy Hansen wrote:
>
> > I have three web servers being load balanced. All the servers are
> > writting their access logs over nfs to the same file. I'm using Linux
> > knfsd with kernel's 2.2.12 from Red Hat 6.1 and kernel's 2.2.11-ac3 on the
> > web servers with nfsv3 client patches from Trond applied.
> >
> > Basically I'm getting very corrupt log files. Half lines, lines running
> > into each other, basically everything is telling me that locking is NOT
> > working.
>
> > So...I don't know what else to explain. Like I said, if there's a better
> > list, please let me know, but I'm going crazy trying to get this going. I
> > have a perl script I wrote to weed out bad entries so we can run our
> > reports, btw WebTrends sucks for its tolerance on log file errors, but I
> > don't know if I'm loosing information or what.
>
>
http://www.xxedgexx.com | jeremy@xxedgexx.com
---------------------------------------------
Y2K. We're all gonna die.
Re: OFF TOPIC: Any general apache administration mail lists?
Posted by Peter W <pe...@usa.net>.
I can understand using nfs to share Web content on identical servers, esp.
if you have a really fast nfs toaster, but why don't you write separate
logs and recombine them before running WebTrends. IIRC, it tolerates
out-of-sequence log entries. If not, you could use a CustomLog format that
adds a timestamp at the beginning of the line, and then do a
merge->sort->trim routine at the end.
-Peter
At 3:43pm Nov 12, 1999, Jeremy Hansen wrote:
> I have three web servers being load balanced. All the servers are
> writting their access logs over nfs to the same file. I'm using Linux
> knfsd with kernel's 2.2.12 from Red Hat 6.1 and kernel's 2.2.11-ac3 on the
> web servers with nfsv3 client patches from Trond applied.
>
> Basically I'm getting very corrupt log files. Half lines, lines running
> into each other, basically everything is telling me that locking is NOT
> working.
> So...I don't know what else to explain. Like I said, if there's a better
> list, please let me know, but I'm going crazy trying to get this going. I
> have a perl script I wrote to weed out bad entries so we can run our
> reports, btw WebTrends sucks for its tolerance on log file errors, but I
> don't know if I'm loosing information or what.
Re: OFF TOPIC: Any general apache administration mail lists?
Posted by Jeremy Hansen <je...@xxedgexx.com>.
Yes! I don't want to use this list for something other then what's
intended, but since you ask :-)
I've actually been working with one of the Linux NFS maintainers, Trond
Myklebust and so far nothing works in my situation. I thought LockFile
and FCNTL in Linux was supposed to do the trick, but well, it's not.
I have three web servers being load balanced. All the servers are
writting their access logs over nfs to the same file. I'm using Linux
knfsd with kernel's 2.2.12 from Red Hat 6.1 and kernel's 2.2.11-ac3 on the
web servers with nfsv3 client patches from Trond applied.
Basically I'm getting very corrupt log files. Half lines, lines running
into each other, basically everything is telling me that locking is NOT
working.
Locking does seem to work fine from test programs Trond gave me, but
Apache is not showing the same information.
My LockFile currently points to a lock disk, /var/run, the default,
although I never actually see a lock file being created. I switched to
USE_FLOCK_blah_blah and then I did see the lock file being created, but
Trond explained to me the flock is a BSD style lock and will not work with
Linux NFS locking, only for local locks, and well it didn't work so...
So...I don't know what else to explain. Like I said, if there's a better
list, please let me know, but I'm going crazy trying to get this going. I
have a perl script I wrote to weed out bad entries so we can run our
reports, btw WebTrends sucks for its tolerance on log file errors, but I
don't know if I'm loosing information or what.
Thanks
-jeremy
> > On Fri, 12 Nov 1999, Jeremy Hansen wrote:
> > > I'm just looking any general apache administration mail lists. I'm having
> > > VERY MANY issues with Apache and nfs, specifically file locking and log
> > > writing via nfs and I'm pretty desperate to figure out what's going on and
> > > what I can do to fix it.
> >
> > Problems that are not solved by putting the lock file on the local disk
> > with:
> >
> > LockFile /some/local/dir/httpd.lock
> >
> > ?
> >
> > -Rasmus
> >
>
>
> http://www.xxedgexx.com | jeremy@xxedgexx.com
> ---------------------------------------------
> Y2K. We're all gonna die.
>
>
>
>
http://www.xxedgexx.com | jeremy@xxedgexx.com
---------------------------------------------
Y2K. We're all gonna die.
Re: OFF TOPIC: Any general apache administration mail lists?
Posted by Jeremy Hansen <je...@xxedgexx.com>.
Yes! I don't want to use this list for something other then what's
intended, but since you ask :-)
I've actually been working with one of the Linux NFS maintainers, Trond
Myklebust and so far nothing works in my situation. I thought LockFile
and FCNTL in Linux was supposed to do the trick, but well, it's not.
I have three web servers being load balanced. All the servers are
writting their access logs over nfs to the same file. I'm using Linux
knfsd with kernel's 2.2.12 from Red Hat 6.1 and kernel's 2.2.11-ac3 on the
web servers with nfsv3 client patches from Trond applied.
Basically I'm getting very corrupt log files. Half lines, lines running
into each other, basically everything is telling me that locking is NOT
working.
Locking does seem to work fine from test programs Trond gave me, but
Apache is not showing the same information.
My LockFile currently points to a lock disk, /var/run, the default,
although I never actually see a lock file being created. I switched to
USE_FLOCK_blah_blah and then I did see the lock file being created, but
Trond explained to me the flock is a BSD style lock and will not work with
Linux NFS locking, only for local locks, and well it didn't work so...
So...I don't know what else to explain. Like I said, if there's a better
list, please let me know, but I'm going crazy trying to get this going. I
have a perl script I wrote to weed out bad entries so we can run our
reports, btw WebTrends sucks for its tolerance on log file errors, but I
don't know if I'm loosing information or what.
Thanks
-jeremy
> On Fri, 12 Nov 1999, Jeremy Hansen wrote:
> > I'm just looking any general apache administration mail lists. I'm having
> > VERY MANY issues with Apache and nfs, specifically file locking and log
> > writing via nfs and I'm pretty desperate to figure out what's going on and
> > what I can do to fix it.
>
> Problems that are not solved by putting the lock file on the local disk
> with:
>
> LockFile /some/local/dir/httpd.lock
>
> ?
>
> -Rasmus
>
http://www.xxedgexx.com | jeremy@xxedgexx.com
---------------------------------------------
Y2K. We're all gonna die.
Re: OFF TOPIC: Any general apache administration mail lists?
Posted by Rasmus Lerdorf <ra...@apache.org>.
On Fri, 12 Nov 1999, Jeremy Hansen wrote:
> I'm just looking any general apache administration mail lists. I'm having
> VERY MANY issues with Apache and nfs, specifically file locking and log
> writing via nfs and I'm pretty desperate to figure out what's going on and
> what I can do to fix it.
Problems that are not solved by putting the lock file on the local disk
with:
LockFile /some/local/dir/httpd.lock
?
-Rasmus
OFF TOPIC: Any general apache administration mail lists?
Posted by Jeremy Hansen <je...@xxedgexx.com>.
I'm just looking any general apache administration mail lists. I'm having
VERY MANY issues with Apache and nfs, specifically file locking and log
writing via nfs and I'm pretty desperate to figure out what's going on and
what I can do to fix it.
Thanks
-jeremy
Re: axe proxy support from 2.0?
Posted by David Reid <ab...@dial.pipex.com>.
Rather than discussing NOT removing something, should we talk about what we
WANT in the new "proxy/file getter/cache/whatever we want to call it"?
Might be more productive and then sets the targets for someone to start...
Just my 2p...
david
----- Original Message -----
From: <tv...@aventail.com>
To: <ne...@apache.org>
Sent: 12 November 1999 18:17
Subject: Re: axe proxy support from 2.0?
> Graham Leggett <mi...@sharp.fm> writes:
>
> > Greg Stein wrote:
> >
> > > All right. Axe the proxy thing. Build in specific support for grabbing
> > > files from elsewhere. If that goes best into the layering, as another
APR
> > > type, or whatever, then it is clear what is going on.
> > >
> > > It seems that we could simplify a lot of our life if we stopped
viewing it
> > > as a (reverse) proxy and viewed/built it as a set of
> > > functions/types/whatever for fetching remote files. Certainly a lot
easier
> > > to build (rather than porting/rebuilding mod_proxy for 2.0).
> >
> > This is true - though this encompasses about 90% of the existing proxy
> > code anyway. Remember that the ftp and http "client" bits as well as the
> > caching engine all have significant use inside the reverse proxy,
> > especially the caching - it makes a huge difference on a busy server.
> > Even the parent cache support is useful to connect the reverse proxy
> > through a further proxy.
>
> The cache is key. We've seen latency times increase by as much as a factor
> of six without it. (The more cache friendly an origin server is, the
> better.) It might be useful to abstract the cache out as well. I suppose
> mod_include might want to cache a document it somehow knows can be cached.
>
> The ability to add other protocols, like ones that start with 'H' and end
> with 'S', would be nice too.
>
> -Tom
>
> --
> Tom Vaughan <tvaughan at aventail dot com>
Re: axe proxy support from 2.0?
Posted by tv...@aventail.com.
Graham Leggett <mi...@sharp.fm> writes:
> Greg Stein wrote:
>
> > All right. Axe the proxy thing. Build in specific support for grabbing
> > files from elsewhere. If that goes best into the layering, as another APR
> > type, or whatever, then it is clear what is going on.
> >
> > It seems that we could simplify a lot of our life if we stopped viewing it
> > as a (reverse) proxy and viewed/built it as a set of
> > functions/types/whatever for fetching remote files. Certainly a lot easier
> > to build (rather than porting/rebuilding mod_proxy for 2.0).
>
> This is true - though this encompasses about 90% of the existing proxy
> code anyway. Remember that the ftp and http "client" bits as well as the
> caching engine all have significant use inside the reverse proxy,
> especially the caching - it makes a huge difference on a busy server.
> Even the parent cache support is useful to connect the reverse proxy
> through a further proxy.
The cache is key. We've seen latency times increase by as much as a factor
of six without it. (The more cache friendly an origin server is, the
better.) It might be useful to abstract the cache out as well. I suppose
mod_include might want to cache a document it somehow knows can be cached.
The ability to add other protocols, like ones that start with 'H' and end
with 'S', would be nice too.
-Tom
--
Tom Vaughan <tvaughan at aventail dot com>
Re: axe proxy support from 2.0?
Posted by Graham Leggett <mi...@sharp.fm>.
Greg Stein wrote:
> It seems that every reply has been "I want its reverse-proxy feature so
> that I can grab files from elsewhere and serve them up as my own server's
> page."
Yep.
> All right. Axe the proxy thing. Build in specific support for grabbing
> files from elsewhere. If that goes best into the layering, as another APR
> type, or whatever, then it is clear what is going on.
>
> It seems that we could simplify a lot of our life if we stopped viewing it
> as a (reverse) proxy and viewed/built it as a set of
> functions/types/whatever for fetching remote files. Certainly a lot easier
> to build (rather than porting/rebuilding mod_proxy for 2.0).
This is true - though this encompasses about 90% of the existing proxy
code anyway. Remember that the ftp and http "client" bits as well as the
caching engine all have significant use inside the reverse proxy,
especially the caching - it makes a huge difference on a busy server.
Even the parent cache support is useful to connect the reverse proxy
through a further proxy.
A reverse proxy is simply the forward proxy engine that responds to a
browser using HTTP instead of proxy-speak (which is similar anyway).
Regards,
Graham
--
-----------------------------------------
minfrin@sharp.fm "There's a moon
over Bourbon Street
tonight...
Re: axe proxy support from 2.0?
Posted by Greg Stein <gs...@lyra.org>.
On Fri, 12 Nov 1999, Graham Leggett wrote:
>...
> Perhaps if the proxy module were to be streamlined, it could become an
> abstraction of the file layer that currently exists in Apache. On a
> typical webserver a normal file and an executable program are two
> sources of data that can be served to clients. The reverse proxy code
> adds HTTP and FTP URLs to these data sources, and are incredibly useful
> for plugging in weird setups into your website that would otherwise
> require messy redirects to accomplish.
It seems that every reply has been "I want its reverse-proxy feature so
that I can grab files from elsewhere and serve them up as my own server's
page."
All right. Axe the proxy thing. Build in specific support for grabbing
files from elsewhere. If that goes best into the layering, as another APR
type, or whatever, then it is clear what is going on.
It seems that we could simplify a lot of our life if we stopped viewing it
as a (reverse) proxy and viewed/built it as a set of
functions/types/whatever for fetching remote files. Certainly a lot easier
to build (rather than porting/rebuilding mod_proxy for 2.0).
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
Re: axe proxy support from 2.0?
Posted by Graham Leggett <mi...@sharp.fm>.
Greg Stein wrote:
> This came up a while back, and there is actually an item in the STATUS
> file, apparently (Manoj said so :-), but why do we still have the proxy
> support in Apache? It seems like it has a largish number of bugs and is
> pretty orthogonal to the rest of Apache. Can't we just axe it and refer
> people to Squid?
>
> It seems like there are sufficient alternatives in the open-source world
> that we shouldn't try to pretend to be proxy developers...
Aaaargh - no! Don't you dare... (much pleading, etc).
The proxy module is one of the primary reasons why we ditched most of
our previous Netscape installation and switched to Apache.
Apache seems to be the only server in existance that builds it's reverse
proxy support into the webserver, rather than hacking reverse proxy
support into forward proxies like NS Proxy server and Squid. This makes
an *enormous* difference, because it means we can use all the good
features of Apache (like split log files, virtual hosts, etc) while at
the same time plugging in all the weird proprietry stuff that we've
inherited, like ASP scripts, Ultraseek search engine, Livewire, etc etc
into our website where required.
As a result we have http://www.ericsson.se served natively by Apache,
http://www.ericsson.se/find served by Ultraseek, and
http://www.ericsson.se/callcentre served by IIS, all under one virtual
host. This is an unmanagable setup using a pure forward proxy like
Netscape Proxy server (we tried it).
Perhaps if the proxy module were to be streamlined, it could become an
abstraction of the file layer that currently exists in Apache. On a
typical webserver a normal file and an executable program are two
sources of data that can be served to clients. The reverse proxy code
adds HTTP and FTP URLs to these data sources, and are incredibly useful
for plugging in weird setups into your website that would otherwise
require messy redirects to accomplish.
Regards,
Graham
--
-----------------------------------------
minfrin@sharp.fm "There's a moon
over Bourbon Street
tonight...
Re: axe proxy support from 2.0?
Posted by Christof Damian <cd...@guideguide.com>.
Greg Stein wrote:
>
> This came up a while back, and there is actually an item in the STATUS
> file, apparently (Manoj said so :-), but why do we still have the proxy
> support in Apache? It seems like it has a largish number of bugs and is
> pretty orthogonal to the rest of Apache. Can't we just axe it and refer
> people to Squid?
I think the proxy support is very useful with ProxyPass and
mod_rewrite, you can do some stuff with this that you can't really do
just with squid & co.
christof
--
Christof Damian [ we are looking for perl and linux
Technical Director programmers, more information @
guideguide ltd. http://mediaconsult.com/wanted ]
Re: axe proxy support from 2.0?
Posted by Manoj Kasichainula <ma...@io.com>.
On Thu, Nov 11, 1999 at 06:01:03PM -0700, Marc Slemko wrote:
> Now, maybe the current proxy should be ripped out and thrown away to give
> someone incentive to implement one that works right and is better
> integrated
This is basically where I stand. I'm not terribly excited about
porting mod_proxy to 2.0.
--
Manoj Kasichainula - manojk at io dot com - http://www.io.com/~manojk/
Re: proxy caching is important
Posted by Brian Behlendorf <br...@apache.org>.
On Wed, 17 Nov 1999, Dean Gaudet wrote:
> in a sense, i envision apache as an "HTTP router". it communicates with
> the client, parses the url, consults its cache and serves from there if
> available, otherwise it passes through to a backend.
I've seen it this way for a long time; I see lots of groups trying to
figure out their own backend protocols as well.
Brian
Re: proxy caching is important
Posted by Tony Finch <do...@dotat.at>.
Graham Leggett <mi...@sharp.fm> wrote:
>
>The problem of checking the whether cached data has changed on disk can
>be handled the same way a normal forward proxy handles it - using data
>aging and shift-reload in the browser to force a cache refresh.
In the situation where we use reverse proxy caches (an almost-entirely
static content server) it's possible to do much better than the usual
"guess and hope we aren't too far wrong" approach to cache freshness.
We know about all the updates to the content on the server and we have
a mechanism for notifying the caches of changes, so when users alter
their pages they see the update immediately without having to fiddle
with shift-reload (which they find quite tricky).
The cacheing mechanism in Apache should aim for that level of service.
Tony.
--
how to dot at
Re: proxy caching is important
Posted by Graham Leggett <mi...@sharp.fm>.
Ryan Bloom wrote:
> Okay, I gave my APR opinion, now for my Apache opinion. (I love changing
> hats :-)
:)
> I don't see a good way to do this without I/O layering, or without being
> able to determine which modules are loaded in the server. I/O layering
> isn't coming until 2.1 at the earliest. That is an upsetting but true
> fact. It is just going to take too long to put it into 2.0, and we'll
> never release a beta.
This is why I would build the hashing functions into either APR or if
not appropriate the Apache core.
The putting-objects-into-cache function would then be built in to the
other modules where needed and practical, knowing that without IO
layering the full solution isn't here yet. For example the module
responsible for shipping static content should be the first place to
build the cache into, what gets pumped to the browser also gets pumped
to the cache. These modules will be using the hash table functions
implemented either in APR or the core, but not in mod_cache (even though
it would be nice if it was).
The pulling-cached-objects-out-and-serving-them job will be done by a
piece of code called mod_cache. It's job is to decide if it's
appropriate to serve cached data (Pragma: no-cache stopping me
perhaps?), then to determine if there is cached data (no there isn't,
I'll get over it) and last of all to ship the data to the client if all
is well.
I get your point about the hashing function, so the underlying "common
code" will be a general hash table accesible to all of Apache-land
either via APR or via the Apache core, whichever. Putting stuff into the
cache will be a function of those areas of the code that have been
modified to support it. To start with simple bits like the serving of
static content only, with more complex bits down the line. Finally the
last third of the job will be handling by mod_cache as described above.
mod_cache would be only 33% of the solution, not 100%.
Regards,
Graham
--
-----------------------------------------
minfrin@sharp.fm "There's a moon
over Bourbon Street
tonight...
Re: proxy caching is important
Posted by Ryan Bloom <rb...@raleigh.ibm.com>.
Okay, I gave my APR opinion, now for my Apache opinion. (I love changing
hats :-)
> Throughout the other modules, content that is shipped to the browser is
> either given to the cache API via ap_cache_in() (if it's static and
> should be cached), or deleted from the cache via ap_cache_del() (if it's
> dynamic and shouldn't be cached, like CGIs).
>
> The mod_cache module would then be responsible for deciding whether it
> should deliver the content from the cache using ap_cache_out().
>
> The problem of checking the whether cached data has changed on disk can
> be handled the same way a normal forward proxy handles it - using data
> aging and shift-reload in the browser to force a cache refresh.
>
> Is this a good way to do this? What do people think?
I don't see a good way to do this without I/O layering, or without being
able to determine which modules are loaded in the server. I/O layering
isn't coming until 2.1 at the earliest. That is an upsetting but true
fact. It is just going to take too long to put it into 2.0, and we'll
never release a beta.
Determining which modules are loaded is easy, and I know how to do it, but
it relies on hash functions, and I haven't had a lot of time recently.
This would allow modules to call ap_is_mod_loaded(char *), where the char
* is the name of the module minus the mod_. It would return TRUE or
FALSE. This would allow mod_cgi to call mod_cache's api's without
worrying about whether or not the module were there. Mod_cgi could
determine if the modules were there. I would also expect modules to cache
the results themselves, not that this owuld be an expensive function.
Just some thoughts.
Ryan
_______________________________________________________________________
Ryan Bloom rbb@raleigh.ibm.com
4205 S Miami Blvd
RTP, NC 27709 It's a beautiful sight to see good dancers
doing simple steps. It's a painful sight to
see beginners doing complicated patterns.
Re: proxy caching is important
Posted by Graham Leggett <mi...@sharp.fm>.
Ryan Bloom wrote:
> Well, APR was mentioned, so I'm jumping in. I would prefer not to use
> ap_cache_(in|out|etc). This is because I think apr needs to be more
> generalized than that. What we really want/need, is a simple hash table.
> The contexts already let us put this into whatever scope we want to put it
> in.
That's pretty much what I was after, which was why I (vaguely) indicated
the "key" in brackets.
The "cache" or "hash table" library's job would be to store binary
objects in a combination of memory and disk, accessed through some kind
of key. That key could be anything, a URL (in our case), a string of
digits (in some-other-app-that-we-haven't-thought-of-yet), some string
basically.
The three functions I listed would be all that's needed - put an object
in the hash table, take one out of the hash table, delete the object
from the hash table, or get info about an object in the hash table (it's
size, mime type, whatever).
> BTW, if the hash table functions aren't in APR already, they will be there
> soon. I am having a hard time keeping up with everything right now. :-)
If the hash table you're thinking of is the same as the one I'm thinking
of, then I thing half the problem will be already solved.
Regards,
Graham
--
-----------------------------------------
minfrin@sharp.fm "There's a moon
over Bourbon Street
tonight...
Re: proxy caching is important
Posted by Dean Gaudet <dg...@arctic.org>.
+1 :)
Dean
On Thu, 18 Nov 1999, Greg Stein wrote:
> If that's your intent, then don't port it. You'll just drag out the
> problem. People get motivated when something doesn't work. :-)
>
> -g
>
> On Thu, 18 Nov 1999, Ryan Bloom wrote:
> > I only volunteered to port the proxy module so we would have a working
> > module while somebody else re-designed the thing. I think re-designing
> > the proxy module is in everybody's best interest, but we need something
> > that works in the mean-time.
> >
> > Ryan
> >
> > On Thu, 18 Nov 1999, Greg Stein wrote:
> > > I volunteered to axe the proxy stuff. Not rewrite it :-)
> > >
> > > It looks like Martin and Ryan have stepped up to at least port the thing
> > > to 2.0. That renders some of my rationale as moot (although I still think
> > > we could reduce our complexity, bug count, maintenance by axing some/all
> > > of the proxy support).
> > >
> > > Cheers,
> > > -g
>
> --
> Greg Stein, http://www.lyra.org/
>
>
Re: proxy caching is important
Posted by Greg Stein <gs...@lyra.org>.
If that's your intent, then don't port it. You'll just drag out the
problem. People get motivated when something doesn't work. :-)
-g
On Thu, 18 Nov 1999, Ryan Bloom wrote:
> I only volunteered to port the proxy module so we would have a working
> module while somebody else re-designed the thing. I think re-designing
> the proxy module is in everybody's best interest, but we need something
> that works in the mean-time.
>
> Ryan
>
> On Thu, 18 Nov 1999, Greg Stein wrote:
> > I volunteered to axe the proxy stuff. Not rewrite it :-)
> >
> > It looks like Martin and Ryan have stepped up to at least port the thing
> > to 2.0. That renders some of my rationale as moot (although I still think
> > we could reduce our complexity, bug count, maintenance by axing some/all
> > of the proxy support).
> >
> > Cheers,
> > -g
--
Greg Stein, http://www.lyra.org/
Re: proxy caching is important
Posted by Ryan Bloom <rb...@raleigh.ibm.com>.
I only volunteered to port the proxy module so we would have a working
module while somebody else re-designed the thing. I think re-designing
the proxy module is in everybody's best interest, but we need something
that works in the mean-time.
Ryan
On Thu, 18 Nov 1999, Greg Stein wrote:
> I volunteered to axe the proxy stuff. Not rewrite it :-)
>
> It looks like Martin and Ryan have stepped up to at least port the thing
> to 2.0. That renders some of my rationale as moot (although I still think
> we could reduce our complexity, bug count, maintenance by axing some/all
> of the proxy support).
>
> Cheers,
> -g
>
>
> On Thu, 18 Nov 1999, David Reid wrote:
>
> > Nice of you to volunteer????
> >
> > d :-)
> > ----- Original Message -----
> > From: Greg Stein <gs...@lyra.org>
> > To: <ne...@apache.org>
> > Sent: 18 November 1999 12:18
> > Subject: Re: proxy caching is important
> >
> >
> > > On Thu, 18 Nov 1999, Ryan Bloom wrote:
> > > > > In terms of the caching itself, how about a generic mod_cache module
> > > > > that handles it?
> > > > >
> > > > > A generic set of caching functions could be added to APR, something
> > > > > like:
> > > >
> > > > Well, APR was mentioned, so I'm jumping in. I would prefer not to use
> > > > ap_cache_(in|out|etc). This is because I think apr needs to be more
> > > > generalized than that. What we really want/need, is a simple hash
> > table.
> > > > The contexts already let us put this into whatever scope we want to put
> > it
> > > > in.
> > > >
> > > > BTW, if the hash table functions aren't in APR already, they will be
> > there
> > > > soon. I am having a hard time keeping up with everything right now. :-)
> > >
> > > That's because something like the caching doesn't go in APR. It is easily
> > > built on *top* of APR. You can keep up if you keep stuff *out* of APR :-)
> > >
> > > And on a separate note: I also believe the HTTP/FTP fetching stuff does
> > > not go into APR either, but into a utility library on top of it.
> > >
> > > Cheers,
> > > -g
> > >
> > > --
> > > Greg Stein, http://www.lyra.org/
> > >
> >
>
> --
> Greg Stein, http://www.lyra.org/
>
_______________________________________________________________________
Ryan Bloom rbb@raleigh.ibm.com
4205 S Miami Blvd
RTP, NC 27709 It's a beautiful sight to see good dancers
doing simple steps. It's a painful sight to
see beginners doing complicated patterns.
Re: proxy caching is important
Posted by Greg Stein <gs...@lyra.org>.
I volunteered to axe the proxy stuff. Not rewrite it :-)
It looks like Martin and Ryan have stepped up to at least port the thing
to 2.0. That renders some of my rationale as moot (although I still think
we could reduce our complexity, bug count, maintenance by axing some/all
of the proxy support).
Cheers,
-g
On Thu, 18 Nov 1999, David Reid wrote:
> Nice of you to volunteer????
>
> d :-)
> ----- Original Message -----
> From: Greg Stein <gs...@lyra.org>
> To: <ne...@apache.org>
> Sent: 18 November 1999 12:18
> Subject: Re: proxy caching is important
>
>
> > On Thu, 18 Nov 1999, Ryan Bloom wrote:
> > > > In terms of the caching itself, how about a generic mod_cache module
> > > > that handles it?
> > > >
> > > > A generic set of caching functions could be added to APR, something
> > > > like:
> > >
> > > Well, APR was mentioned, so I'm jumping in. I would prefer not to use
> > > ap_cache_(in|out|etc). This is because I think apr needs to be more
> > > generalized than that. What we really want/need, is a simple hash
> table.
> > > The contexts already let us put this into whatever scope we want to put
> it
> > > in.
> > >
> > > BTW, if the hash table functions aren't in APR already, they will be
> there
> > > soon. I am having a hard time keeping up with everything right now. :-)
> >
> > That's because something like the caching doesn't go in APR. It is easily
> > built on *top* of APR. You can keep up if you keep stuff *out* of APR :-)
> >
> > And on a separate note: I also believe the HTTP/FTP fetching stuff does
> > not go into APR either, but into a utility library on top of it.
> >
> > Cheers,
> > -g
> >
> > --
> > Greg Stein, http://www.lyra.org/
> >
>
--
Greg Stein, http://www.lyra.org/
Re: proxy caching is important
Posted by David Reid <ab...@dial.pipex.com>.
Nice of you to volunteer????
d :-)
----- Original Message -----
From: Greg Stein <gs...@lyra.org>
To: <ne...@apache.org>
Sent: 18 November 1999 12:18
Subject: Re: proxy caching is important
> On Thu, 18 Nov 1999, Ryan Bloom wrote:
> > > In terms of the caching itself, how about a generic mod_cache module
> > > that handles it?
> > >
> > > A generic set of caching functions could be added to APR, something
> > > like:
> >
> > Well, APR was mentioned, so I'm jumping in. I would prefer not to use
> > ap_cache_(in|out|etc). This is because I think apr needs to be more
> > generalized than that. What we really want/need, is a simple hash
table.
> > The contexts already let us put this into whatever scope we want to put
it
> > in.
> >
> > BTW, if the hash table functions aren't in APR already, they will be
there
> > soon. I am having a hard time keeping up with everything right now. :-)
>
> That's because something like the caching doesn't go in APR. It is easily
> built on *top* of APR. You can keep up if you keep stuff *out* of APR :-)
>
> And on a separate note: I also believe the HTTP/FTP fetching stuff does
> not go into APR either, but into a utility library on top of it.
>
> Cheers,
> -g
>
> --
> Greg Stein, http://www.lyra.org/
>
Re: proxy caching is important
Posted by Ryan Bloom <rb...@raleigh.ibm.com>.
> > BTW, if the hash table functions aren't in APR already, they will be there
> > soon. I am having a hard time keeping up with everything right now. :-)
>
> That's because something like the caching doesn't go in APR. It is easily
> built on *top* of APR. You can keep up if you keep stuff *out* of APR :-)
>
> And on a separate note: I also believe the HTTP/FTP fetching stuff does
> not go into APR either, but into a utility library on top of it.
That's exactly my point though. The cache was to specific. What belongs
in APR is the hash functions, nothing more. What belongs in mod_cache is
the add_cache/remove_cache functions. Those functions USE the APR hash
functions.
I have been trying very hard not to put things into APR. I do not want to
tie APR to Apache in any way. The more cruft we put into APR that makes
it a web server library, the less likely other people are to use it. If I
am not doing a good job of keeping things out of APR, let me know. I will
struggle to fix it. :-)
Ryan
_______________________________________________________________________
Ryan Bloom rbb@raleigh.ibm.com
4205 S Miami Blvd
RTP, NC 27709 It's a beautiful sight to see good dancers
doing simple steps. It's a painful sight to
see beginners doing complicated patterns.
Re: proxy caching is important
Posted by Greg Stein <gs...@lyra.org>.
On Thu, 18 Nov 1999, Ryan Bloom wrote:
> > In terms of the caching itself, how about a generic mod_cache module
> > that handles it?
> >
> > A generic set of caching functions could be added to APR, something
> > like:
>
> Well, APR was mentioned, so I'm jumping in. I would prefer not to use
> ap_cache_(in|out|etc). This is because I think apr needs to be more
> generalized than that. What we really want/need, is a simple hash table.
> The contexts already let us put this into whatever scope we want to put it
> in.
>
> BTW, if the hash table functions aren't in APR already, they will be there
> soon. I am having a hard time keeping up with everything right now. :-)
That's because something like the caching doesn't go in APR. It is easily
built on *top* of APR. You can keep up if you keep stuff *out* of APR :-)
And on a separate note: I also believe the HTTP/FTP fetching stuff does
not go into APR either, but into a utility library on top of it.
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
Re: proxy caching is important
Posted by Ryan Bloom <rb...@raleigh.ibm.com>.
> In terms of the caching itself, how about a generic mod_cache module
> that handles it?
>
> A generic set of caching functions could be added to APR, something
> like:
Well, APR was mentioned, so I'm jumping in. I would prefer not to use
ap_cache_(in|out|etc). This is because I think apr needs to be more
generalized than that. What we really want/need, is a simple hash table.
The contexts already let us put this into whatever scope we want to put it
in.
BTW, if the hash table functions aren't in APR already, they will be there
soon. I am having a hard time keeping up with everything right now. :-)
Ryan
_______________________________________________________________________
Ryan Bloom rbb@raleigh.ibm.com
4205 S Miami Blvd
RTP, NC 27709 It's a beautiful sight to see good dancers
doing simple steps. It's a painful sight to
see beginners doing complicated patterns.
Re: proxy caching is important
Posted by Graham Leggett <mi...@sharp.fm>.
Eric Robibaro wrote:
> > my personal opinion is that the caching function should be well integrated
> > with the core, and be generic enough to support multiple methods of
> > populating the cache.
In terms of the caching itself, how about a generic mod_cache module
that handles it?
A generic set of caching functions could be added to APR, something
like:
ap_cache_in(key)
ap_cache_out(key)
ap_cache_del(key)
A combination of memory (MM module?) or disk could be used for the cache
objects.
Throughout the other modules, content that is shipped to the browser is
either given to the cache API via ap_cache_in() (if it's static and
should be cached), or deleted from the cache via ap_cache_del() (if it's
dynamic and shouldn't be cached, like CGIs).
The mod_cache module would then be responsible for deciding whether it
should deliver the content from the cache using ap_cache_out().
The problem of checking the whether cached data has changed on disk can
be handled the same way a normal forward proxy handles it - using data
aging and shift-reload in the browser to force a cache refresh.
Is this a good way to do this? What do people think?
Regards,
Graham
--
-----------------------------------------
minfrin@sharp.fm "There's a moon
over Bourbon Street
tonight...
Re: proxy caching is important
Posted by Eric Robibaro <eh...@point-net.com>.
<snip>
> my personal opinion is that the caching function should be well integrated
> with the core, and be generic enough to support multiple methods of
> populating the cache.
If you'll allow someone who hasn't had time to contribute to the codebase
yet, "I like the way you think"
> in a sense, i envision apache as an "HTTP router". it communicates with
> the client, parses the url, consults its cache and serves from there if
> available, otherwise it passes through to a backend.
>
> notice that serving static content and serving from a cache are
> essentially identical. the only difference is the url->filename map.
I've always found odd how closely mod_rewrite and mod_proxy integrated
each other, could the answer be found by generalizing mod_rewrite to
include the proxy "functions" e.g. the proper apr calls to perform the
url->filename mapping?
Perhaps by joining mod_alias, mod_rewrite and mod_proxy into a sort of
supermodule?
say mod_uri
which would allow uri rewriting, proxying, etc...
perhaps using a different way of specifying rewrites than the (to many)
mod_rewrite rewrites?
> Dean
>
>
--
Eric Robibaro - Administrateur de Systemes - Systems Administrator -
Point Net Connect 4535, Park Avenue, Montreal, Quebec h2v 4e4
May the system be with you ;o)
proxy caching is important
Posted by Dean Gaudet <dg...@arctic.org>.
first off i'll ignore the case of using proxy caches for firewalls... that
is well served by squid, and has somewhat different requirements.
but the case of using HTTP proxying to hide other (usually dynamic
content) webservers behind a single url-space, this is really interesting.
it's actually just one case of the general problem of hiding any dynamic
content engine in the url-space.
much "dynamic" content is actually static, and could take advantage of
caching. caching content is the same whether it gets to the server via
HTTP or some other protocol (such as IPC with another process running a
JVM or perl or php).
my personal opinion is that the caching function should be well integrated
with the core, and be generic enough to support multiple methods of
populating the cache.
in a sense, i envision apache as an "HTTP router". it communicates with
the client, parses the url, consults its cache and serves from there if
available, otherwise it passes through to a backend.
notice that serving static content and serving from a cache are
essentially identical. the only difference is the url->filename map.
Dean
Re: axe proxy support from 2.0?
Posted by tv...@aventail.com.
Marc Slemko <ma...@znep.com> writes:
> On Thu, 11 Nov 1999, Greg Stein wrote:
>
> > This came up a while back, and there is actually an item in the STATUS
> > file, apparently (Manoj said so :-), but why do we still have the proxy
> > support in Apache? It seems like it has a largish number of bugs and is
> > pretty orthogonal to the rest of Apache. Can't we just axe it and refer
> > people to Squid?
> >
> > It seems like there are sufficient alternatives in the open-source world
> > that we shouldn't try to pretend to be proxy developers...
>
> Except that a proxy fits in with a web server very very usefully for a lot
> of things. It lets you fetch various content from various remote
> pages. It lets you do SSIs of remote files (with a two line
> hack). etc. Those things can't be done with an external proxy. In fact,
> I would think it would be useful to expand the idea of the proxy to
> include a better specified interface that lets modules say "gimme this
> page".
>
> Now, maybe the current proxy should be ripped out and thrown away to give
> someone incentive to implement one that works right and is better
> integrated (which doesn't necessarily mean more closely integrated, since
> tight integration can be a very negative thing, but just integrated using
> a better defined interface).
>
> There are a lot of people using Apache's proxy because it is easy to setup
> and provides them with an integrated solution. Something like squid can
> be quite... a handful.
If I may be so bold...
I happen to be one of these people. Although my main interest in the proxy
module is as a reverse proxy. I'd hate to see the proxy module get the
axe. I do have some ideas about what functionality I'd like to see in the
proxy module FWIW.
o It would be nice if the proxy module allowed other modules to
filter resources, via EAPI or some other hook mechanism. This would
allow a module to filter out ActiveX controls on inbound requests,
or perhaps mod_rewrite could rewrite URLs on outbound requests so
that the URLs fall under some ProxyPassReverse namespace.
Currently the proxy module writes data out to the client and to
the cache at the same time. When a resource is filtered, the proxy
module would have to wait until the entire resource is processed
so, for example, Content-Length could be adjusted accordingly.
o It would be nice if the cache replacement algorithm allowed for
"weights", via EAPI or some other hook mechanism. This would allow
filtered resources to be given preference over non-filtered
resources.
o It would be nice to be able to plug-in other cache backends. Like
a shared memory cache a la mod_ssl's shared memory SSL session
cache.
o A cached object currently consists of both the HTTP header and
the data. It would be nice if this was broken up. This would allow
the header to be stored in some optimized binary fashion.
Not that I can promise I'd be able to help. Though I'd like to. But
whomever does improve the proxy module, if anyone, I'd appreciate it if
these ideas were considered at the very least.
Thanks,
Tom
--
Tom Vaughan <tvaughan at aventail dot com>
Re: axe proxy support from 2.0?
Posted by Marc Slemko <ma...@znep.com>.
On Thu, 11 Nov 1999, Greg Stein wrote:
> This came up a while back, and there is actually an item in the STATUS
> file, apparently (Manoj said so :-), but why do we still have the proxy
> support in Apache? It seems like it has a largish number of bugs and is
> pretty orthogonal to the rest of Apache. Can't we just axe it and refer
> people to Squid?
>
> It seems like there are sufficient alternatives in the open-source world
> that we shouldn't try to pretend to be proxy developers...
Except that a proxy fits in with a web server very very usefully for a lot
of things. It lets you fetch various content from various remote
pages. It lets you do SSIs of remote files (with a two line
hack). etc. Those things can't be done with an external proxy. In fact,
I would think it would be useful to expand the idea of the proxy to
include a better specified interface that lets modules say "gimme this
page".
Now, maybe the current proxy should be ripped out and thrown away to give
someone incentive to implement one that works right and is better
integrated (which doesn't necessarily mean more closely integrated, since
tight integration can be a very negative thing, but just integrated using
a better defined interface).
There are a lot of people using Apache's proxy because it is easy to setup
and provides them with an integrated solution. Something like squid can
be quite... a handful.