File upload gateway (was Re: Forking off lengthy processes)

[Jeremy Howard <jh...@fastmail.fm>]

I said:
> > The other place I'd like to avoid holding up my mod_perl processes is in
> > waiting for file uploads to finish (which is a common situation as people
> > upload attachments for their emails). Is there a any way to do this...
> > perhaps by using some kind of 'gateway' server?

Barrie Slaymaker wrote:
> The most often wished-for approach is for a http cache to accept the whole
> request before passing it through to the main httpd, but I'm not aware of
> any that do that.
> 
Exactly! If anyone is doing something similar to this could you direct me
to the code you use? The beauty of it would be that the upload (and other
CGI params) could be passed from the cache to the mod_perl server over
the fast internal network link (so the mod_perl process doesn't wait
long), while the slow upload from a user's modem only holds up a small
httpd process.

I guess this really should be a module written in C, since that would be
the smallest way of doing it memory-wise (I'd rule out a Perl CGI since
it has to be recompiled each time, and this gateway would presumably be
high volume).

How do proxy servers like Squid work? Do they just pass the packets on,
or do they accept the whole request and _then_ pass it on?

-- 
  Jeremy Howard
  jeremy@fastmail.fm

Re: File upload gateway (was Re: Forking off lengthy processes)

[Barrie Slaymaker <rb...@telerama.com>]

> I said:
> >
> > The most often wished-for approach is for a http cache to accept the whole
> > request before passing it through to the main httpd, but I'm not aware of
> > any that do that.
> >

Jeremy Howard wrote:
>
> How do proxy servers like Squid work? 

Squid, Apache in cache mode, et al, was what I meant by (sic) "a http cache", and
I think they pass on chunks of the request as they receive them.  Might be worth
a quick scan of the source code.  One of the problems is that you can't guarantee
to cache all of the upload POST request in memory, given that the size of the
uploads may be huge.  All but the most ancient squids deal with this in the
download direction, I don't know what they do in the upload direction, but my
guess is immediate pass-through, not store & forward (which is what you want) in
order to reduce latency.

- Barrie