Large file support in 2.0.56?

[Brandon Fosdick <bf...@bfoz.net>]

I might have asked this before, but I've forgotten the answer, and so has google. Has any of the large file goodness from 2.2.x made it into 2.0.x? Will it ever?

Re: Large file support in 2.0.56?

[Joe Orton <jo...@redhat.com>]

On Mon, Apr 17, 2006 at 02:40:13PM +0100, Colm MacCarthaigh wrote:
> On Mon, Apr 17, 2006 at 09:09:12AM -0400, Jeff Trawick wrote:
> > On 4/15/06, Brandon Fosdick <bf...@bfoz.net> wrote:
> > > I might have asked this before, but I've forgotten the answer, and so has google. Has any of the large file goodness from 2.2.x made it into 2.0.x? Will it ever?
> > 
> > Different answer than you got before, but I think this is more accurate (Joe?):
> > 
> > Turn on your OS's large file flags in CFLAGS
> > 
> > make distclean && CFLAGS=-D_something ./configure

Specifically:

  CFLAGS="-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64" ./configure

should work with recent 2.0.x releases, in limited circumstances, may 
break with third-party modules, is not generally recommended, and when 
it breaks you get to keep the pieces, etc.  In particular this does not 
support >2Gb request bodies, which it sounds like Brandon wants; you 
really do need 2.2.x for that.

> That works, but does need Joe's split patch;
> 
> 	http://people.apache.org/~jorton/ap_splitlfs.diff

That patch is actually in 2.0.53 and later.

Regards,

joe

Re: Large file support in 2.0.56?

[Colm MacCarthaigh <co...@stdlib.net>]

On Mon, Apr 17, 2006 at 09:09:12AM -0400, Jeff Trawick wrote:
> On 4/15/06, Brandon Fosdick <bf...@bfoz.net> wrote:
> > I might have asked this before, but I've forgotten the answer, and so has google. Has any of the large file goodness from 2.2.x made it into 2.0.x? Will it ever?
> 
> Different answer than you got before, but I think this is more accurate (Joe?):
> 
> Turn on your OS's large file flags in CFLAGS
> 
> make distclean && CFLAGS=-D_something ./configure

That works, but does need Joe's split patch;

	http://people.apache.org/~jorton/ap_splitlfs.diff

:)

-- 
Colm MacCárthaigh                        Public Key: colm+pgp@stdlib.net

Re: Large file support in 2.0.56?

[Jeff Trawick <tr...@gmail.com>]

On 4/15/06, Brandon Fosdick <bf...@bfoz.net> wrote:
> I might have asked this before, but I've forgotten the answer, and so has google. Has any of the large file goodness from 2.2.x made it into 2.0.x? Will it ever?

Different answer than you got before, but I think this is more accurate (Joe?):

Turn on your OS's large file flags in CFLAGS

make distclean && CFLAGS=-D_something ./configure

and you get the support. This isn't the default with APR 0.9.x (and
thus Apache httpd 2.0.x) because it breaks binary compatibility with
existing builds.  As long as you use only modules that you can
recompile and don't have bugs exposed only with large file support
enabled you should be okay.

Re: Large file support in 2.0.56?

[Brandon Fosdick <bf...@bfoz.net>]

Brandon Fosdick wrote:
 > It does seem like a rather small and arbitrary limit. I can't think of 
> what else besides apache would cause it, but I could be missing 
> something. The files are being dumped into mysql in 64K blocks. The 
> machine is an amd64, so that shouldn't be a problem, and 700MB isn't 
> near 2 or 4 GB anyway. Uploading from a cable modem doesn't go anywhere 
> near saturating the disk, cpu, or network. I've tried OSX, Win2k and 
> WinXP, all with the same result. I'm running out of things to check. Any 
> suggestions? I guess it could be a limit in mod_dav itself. I'm afraid 
> to go there...it looks messy.


In the interest of posterity and curiosity I played with 2.0.55 a little bit more, and I thought I'd document it here in case anyone cares.

My test setup includes a server running FreeBSD 6.1-PRE on a Sempron 3100+ and a PowerBookG4 (Tiger 10.4.5) acting as the client. I wrote a mod_dav storage provider that uses mysql as a back end. Files are stored as a series of 64K records.

Previously I had reported upload failure of files greater than ~700MB. On a hunch I took half the RAM out of the server, just to see if it's a memory problem. Sure enough, the limit appears to be in the 300MB - 400MB range now (specifically, a win2k iso). But that still seemed odd, since from previous testing I had determined that httpd/mod_dav streams the files to my provider in 2K chunks, which are then assembled into 64K chunks and written to the database. It didn't seem likely that I had enough bytes buffered at any one time to cause memory starvation. Plus, I have more than enough swap configured.

So I redid the test with top running (for lack of a better idea) and a debug entry going to the error log for each chunk of bytes given to the storage provider. Bytes are still being provided in 2K chunks. I observed the following sequence:

1. OSX transfer window opens, progress bar proceeds rapidly to completion, and then says "closing file", which it continues to say throughout the transfer. httpd and mysqld are using 0% of the CPU and 0% of disk i/o. No log entries.

2. httpd jumps to using ~20% of the CPU and 100% of disk i/o. Debug log entries begin. Entries report 2K chunks.

3. 1-2 minutes later, OSX reports "server disconnected". mod_dav is still streaming 2K chunks, httpd is still using 100% disk i/o.

4. 1-2 minutes later, httpd client appears to crash and respawn. It appears to either restart the transfer or re-stream the file to storage, but at this point the log file becomes corrupt.

5. If I cancel the transfer at this point at least one of the httpd processes starts using 100% CPU and 100% disk i/o. Then I killall httpd.

For completeness, I re-ran the above test without the debug logging and w/o mod_security and observed the same behavior.

At this point I'm not sure if I should bother trying the large file hack for 2.0.55 or just start migrating to 2.2.x. This no longer seems to be a large file problem, but I'm not sure what kind of problem it is. Judging by the mod_security audit log, the client doesn't appear to be doing anything odd. That is, it's just issuing a LOCK and a PUT for each file upload. Naturally, my own code is suspect as well, but it never sees anything bigger than 64K, and it can handle larger files when I haven't taken half the RAM out.

I'm stumped.

Unless anyone has a better idea I'll just give 2.2 a try and see what happens.

Re: Large file support in 2.0.56?

["William A. Rowe, Jr." <wr...@rowe-clan.net>]

Brandon Fosdick wrote:
> Nick Kew wrote:
> 
>> I haven't tried files that size, but that's far too small for 
>> LARGE_FILE to
>> be relevant.  I guess you knew that already, so does something else
>> lead you to suppose you're hitting an Apache limit?
> 
> It does seem like a rather small and arbitrary limit. I can't think of 
> what else besides apache would cause it, but I could be missing 
> something. The files are being dumped into mysql in 64K blocks. The 
> machine is an amd64, so that shouldn't be a problem, and 700MB isn't 
> near 2 or 4 GB anyway. Uploading from a cable modem doesn't go anywhere 
> near saturating the disk, cpu, or network. I've tried OSX, Win2k and 
> WinXP, all with the same result. I'm running out of things to check. Any 
> suggestions? I guess it could be a limit in mod_dav itself. I'm afraid 
> to go there...it looks messy.

Well, the content length is stored as an int in httpd 2.0, so that also is
an issue (dav has so much metadata I've no idea how much real data can be
put up to the server.)

Keep in mind that you are gonna hit client bugs as well ;-)  At least httpd
version 2.2 should give you a good baseline to seperate the client from the
server issues.

Re: Large file support in 2.0.56?

[Brandon Fosdick <bf...@bfoz.net>]

Nick Kew wrote:
> I haven't tried files that size, but that's far too small for LARGE_FILE to
> be relevant.  I guess you knew that already, so does something else
> lead you to suppose you're hitting an Apache limit?

It does seem like a rather small and arbitrary limit. I can't think of what else besides apache would cause it, but I could be missing something. The files are being dumped into mysql in 64K blocks. The machine is an amd64, so that shouldn't be a problem, and 700MB isn't near 2 or 4 GB anyway. Uploading from a cable modem doesn't go anywhere near saturating the disk, cpu, or network. I've tried OSX, Win2k and WinXP, all with the same result. I'm running out of things to check. Any suggestions? I guess it could be a limit in mod_dav itself. I'm afraid to go there...it looks messy.


> Regarding your custom AAA module, it should be possible to use that
> in 2.2.  If you don't want to port it, you can fall back to the 2.0
> architecture.  At worst you may have to use 2.0 mod_auth and/or
> mod_access alongside your module.

Ah, that I didn't know. Thanks.

Re: Large file support in 2.0.56?

[Nick Kew <ni...@webthing.com>]

On Sunday 16 April 2006 20:41, Brandon Fosdick wrote:
> William A. Rowe, Jr. wrote:
> > Actually, not entirely true.  There is some chewy goodness now in 2.0.x,
> > such as log files which can grow beyond 2GB, from an APR 0.9
> > APR_LARGE_FILE hack.  It's a gross hack, which means we can't really
> > provide all sorts of large file manipulations, but logging, for example,
> > is one of the most common
> > complaints.
>
> hmmm...that doesn't help me much. I'm more interested in large files in
> mod_dav. Right now I can't upload anything much bigger than 700MB.

I haven't tried files that size, but that's far too small for LARGE_FILE to
be relevant.  I guess you knew that already, so does something else
lead you to suppose you're hitting an Apache limit?

Regarding your custom AAA module, it should be possible to use that
in 2.2.  If you don't want to port it, you can fall back to the 2.0
architecture.  At worst you may have to use 2.0 mod_auth and/or
mod_access alongside your module.

-- 
Nick Kew

Re: Large file support in 2.0.56?

[Olaf van der Spek <ol...@gmail.com>]

On 4/16/06, Joost de Heer <jo...@sanguis.xs4all.nl> wrote:
> Colm MacCarthaigh wrote:
> > On Sun, Apr 16, 2006 at 10:28:10PM +0200, Joost de Heer wrote:
> >>> hmmm...that doesn't help me much. I'm more interested in large files in
> >>> mod_dav. Right now I can't upload anything much bigger than 700MB.
> >> IMO, that's not something a webserver should be used for anyway.
> >
> > I do it all of the time. We have users who upload DVD iso's to their
> > DAV shares. Can't see any reason why DAV shouldn't be capable of such
> > things.
>
> There's a difference between 'being capable of' and 'being the proper tool for'.

Of course, but why wouldn't DAV be the proper tool for it?

Re: Large file support in 2.0.56?

[Joost de Heer <jo...@sanguis.xs4all.nl>]

Colm MacCarthaigh wrote:
> On Sun, Apr 16, 2006 at 10:28:10PM +0200, Joost de Heer wrote:
>>> hmmm...that doesn't help me much. I'm more interested in large files in 
>>> mod_dav. Right now I can't upload anything much bigger than 700MB.
>> IMO, that's not something a webserver should be used for anyway.
> 
> I do it all of the time. We have users who upload DVD iso's to their
> DAV shares. Can't see any reason why DAV shouldn't be capable of such
> things.

There's a difference between 'being capable of' and 'being the proper tool for'.

Joost

Re: Large file support in 2.0.56?

[Colm MacCarthaigh <co...@stdlib.net>]

On Sun, Apr 16, 2006 at 10:28:10PM +0200, Joost de Heer wrote:
> >hmmm...that doesn't help me much. I'm more interested in large files in 
> >mod_dav. Right now I can't upload anything much bigger than 700MB.
> 
> IMO, that's not something a webserver should be used for anyway.

I do it all of the time. We have users who upload DVD iso's to their
DAV shares. Can't see any reason why DAV shouldn't be capable of such
things.

-- 
Colm MacCárthaigh                        Public Key: colm+pgp@stdlib.net

Re: Large file support in 2.0.56?

[Joost de Heer <jo...@sanguis.xs4all.nl>]

> hmmm...that doesn't help me much. I'm more interested in large files in 
> mod_dav. Right now I can't upload anything much bigger than 700MB.

IMO, that's not something a webserver should be used for anyway.

Joost

Re: Large file support in 2.0.56?

[Brandon Fosdick <bf...@bfoz.net>]

William A. Rowe, Jr. wrote:
> Actually, not entirely true.  There is some chewy goodness now in 2.0.x,
> such as log files which can grow beyond 2GB, from an APR 0.9 APR_LARGE_FILE
> hack.  It's a gross hack, which means we can't really provide all sorts of
> large file manipulations, but logging, for example, is one of the most 
> common
> complaints.

hmmm...that doesn't help me much. I'm more interested in large files in mod_dav. Right now I can't upload anything much bigger than 700MB.

> But any further work for Large Files in 2.0 is long abandoned for 
> getting it
> right in the first place, for 2.2.

Oh well. I guess I better go look at the auth stuff again. Thanks.

Re: Large file support in 2.0.56?

["William A. Rowe, Jr." <wr...@rowe-clan.net>]

Paul Querna wrote:
> Brandon Fosdick wrote:
> 
>> I might have asked this before, but I've forgotten the answer, and so 
>> has google. Has any of the large file goodness from 2.2.x made it into 
>> 2.0.x?
> 
> no.

Actually, not entirely true.  There is some chewy goodness now in 2.0.x,
such as log files which can grow beyond 2GB, from an APR 0.9 APR_LARGE_FILE
hack.  It's a gross hack, which means we can't really provide all sorts of
large file manipulations, but logging, for example, is one of the most common
complaints.

But any further work for Large Files in 2.0 is long abandoned for getting it
right in the first place, for 2.2.

Re: Large file support in 2.0.56?

[Brandon Fosdick <bf...@bfoz.net>]

Paul Querna wrote:
> Is there a specific reason you can't use 2.2.x?

AAA screwiness. I ended up writing a custom auth module for 2.0.x, and last time I looked at porting it to 2.2.x my head nearly exploded. And, it seemed like there were still some changes in the works. Has all of that settled down yet?

Re: Large file support in 2.0.56?

[Paul Querna <ch...@force-elite.com>]

Brandon Fosdick wrote:
> I might have asked this before, but I've forgotten the answer, and so 
> has google. Has any of the large file goodness from 2.2.x made it into 
> 2.0.x?

no.

> Will it ever?

no.

Several of the things require APR 1.x, and some of them break binary 
compat.  They will never be fixed in 2.0.x.

Is there a specific reason you can't use 2.2.x?

-Paul