You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Cliff Woolley <jw...@virginia.edu> on 2003/02/28 20:23:31 UTC
apr_lstat() fails on linux for large files
I was trying to track down the cause of PR#17357, and came across
something interesting-slash-annoying.
mod_autoindex in Apache loops over the directory entries it's indexing
using apr_dir_read(). Among other things, apr_dir_read() does an
apr_lstat() call (which in turn calls apr_stat() with the APR_FINFO_LINK
wanted flag) on the next entry in the directory. If you have a large file
(>2GB) in the directory, though, on linux the lstat() will fail with
errno==EOVERFLOW (75) (Value too large for defined data type). This
causes the apr_lstat() to fail which causes the apr_dir_read() to fail
which makes mod_autoindex think it's hit the end of the directory.
Of course, if I go and explictly compile Apache with
CFLAGS=-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64, then it works.
So I guess the ultimate questions are these:
(a) Why doesn't Apache automatically include largefile support?
Performance reasons? Shouldn't there at least be an easy ./configure
option to turn it on?
(b) If it's not turned on, shouldn't mod_autoindex detect the EOVERFLOW
return code and special-case it?
--Cliff
http://nagoya.apache.org/bugzilla/show_bug.cgi?id=17357
Re: apr_lstat() fails on linux for large files
Posted by "William A. Rowe, Jr." <wr...@rowe-clan.net>.
At 05:10 PM 2/28/2003, Joe Orton wrote:
>Yeah, something like that is possible: if you build with
>-D_LARGEFILE_SOURCE you get the "transitional" LFS API which gives you
>open64 (aka passing O_LARGEFILE to open) and lseek64 and things like
>that which work on a new off64_t type, without changing the size of
>off_t.
Ewww... yet another nice reason for never trusting off_t and only relying
on our apr_off_t :-)
Bill
Re: apr_lstat() fails on linux for large files
Posted by Joe Orton <jo...@redhat.com>.
On Fri, Feb 28, 2003 at 04:37:38PM -0500, Jeff Trawick wrote:
> Joe Orton wrote:
> >On Fri, Feb 28, 2003 at 02:23:31PM -0500, Cliff Woolley wrote:
> >...
> >
> >>Of course, if I go and explictly compile Apache with
> >>CFLAGS=-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64, then it works.
> >>
> >>So I guess the ultimate questions are these:
> >>
> >>(a) Why doesn't Apache automatically include largefile support?
> >>Performance reasons? Shouldn't there at least be an easy ./configure
> >>option to turn it on?
> >
> >
> >There are at least two ramifications of doing this (this is all only
> >relevant on 32-bit systems):
>
> isn't there also the problem that sendfile will be broken on many Linux
> boxes?
True, I forgot that.
> >- Apache/APR compiled with _FILE_OFFSET_BITS=64 has an incompatible
> >module ABI with one which isn't - this is especially nasty if you don't
> >detect it at module load time.
>
> something we can fix for APR 1.0/Apache 2.1-dev by making apr_off_t
> always 64 bits whether native apr_off_t is 64 bits or not?
Yeah, something like that is possible: if you build with
-D_LARGEFILE_SOURCE you get the "transitional" LFS API which gives you
open64 (aka passing O_LARGEFILE to open) and lseek64 and things like
that which work on a new off64_t type, without changing the size of
off_t.
joe
Re: apr_lstat() fails on linux for large files
Posted by Wan-Teh Chang <wt...@netscape.com>.
Justin Erenkrantz wrote:
>
> > something we can fix for APR 1.0/Apache 2.1-dev by making apr_off_t always
> > 64 bits whether native apr_off_t is 64 bits or not?
>
> Perhaps that might not be a bad idea, but it would make off_t usage that much
> worse on 32-bit processors and we'd have to downcast apr_off_t to any system
> calls.
I would not worry about these two problems. off_t usage should not be critical to
the performance of your code.
Wan-Teh
Re: apr_lstat() fails on linux for large files
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
[ Moving to dev@apr ]
--On Friday, February 28, 2003 4:37 PM -0500 Jeff Trawick
<tr...@attglobal.net> wrote:
>> > (a) Why doesn't Apache automatically include largefile support?
>> > Performance reasons? Shouldn't there at least be an easy ./configure
>> > option to turn it on?
> isn't there also the problem that sendfile will be broken on many Linux
> boxes?
Yes, Sascha and I had a thread a few weeks ago about this. It seems that
brand new glibc's (2.3+) allow for sendfile() and largefiles. But, anything
not bleeding edge new will cause a compilation fault if we enable largefiles
and use sendfile.
I believe Sascha posted some proof-of-concept code that demonstrated turning
this on at run-time via dlsym tricks. (Not sure I'd like doing dlsym tricks
to solve this, but that's me.)
Linux was the reason I never checked in my largefiles patch - I believe there
is an entry in STATUS for this.
> something we can fix for APR 1.0/Apache 2.1-dev by making apr_off_t always
> 64 bits whether native apr_off_t is 64 bits or not?
Perhaps that might not be a bad idea, but it would make off_t usage that much
worse on 32-bit processors and we'd have to downcast apr_off_t to any system
calls. So, I'd be cautious about doing this. -- justin
Re: apr_lstat() fails on linux for large files
Posted by Jeff Trawick <tr...@attglobal.net>.
William A. Rowe, Jr. wrote:
> At 03:37 PM 2/28/2003, Jeff Trawick wrote:
>
> >something we can fix for APR 1.0/Apache 2.1-dev by making apr_off_t
> always 64 bits
>
>
> +1
>
>
> >whether native apr_off_t is 64 bits or not?
>
>
> s/apr_off_t/off_t/ :-)
oops :) yes, of course that was the intention
Re: apr_lstat() fails on linux for large files
Posted by Jeff Trawick <tr...@attglobal.net>.
William A. Rowe, Jr. wrote:
> At 03:37 PM 2/28/2003, Jeff Trawick wrote:
>
> >something we can fix for APR 1.0/Apache 2.1-dev by making apr_off_t
> always 64 bits
>
>
> +1
>
>
> >whether native apr_off_t is 64 bits or not?
>
>
> s/apr_off_t/off_t/ :-)
oops :) yes, of course that was the intention
Re: apr_lstat() fails on linux for large files
Posted by "William A. Rowe, Jr." <wr...@rowe-clan.net>.
At 03:37 PM 2/28/2003, Jeff Trawick wrote:
>Joe Orton wrote:
>
>>On Fri, Feb 28, 2003 at 02:23:31PM -0500, Cliff Woolley wrote:
>>...
>>
>>>Of course, if I go and explictly compile Apache with
>>>CFLAGS=-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64, then it works.
>>>
>>>So I guess the ultimate questions are these:
>>>
>>>(a) Why doesn't Apache automatically include largefile support?
>>>Performance reasons? Shouldn't there at least be an easy ./configure
>>>option to turn it on?
>>
>>
>>There are at least two ramifications of doing this (this is all only
>>relevant on 32-bit systems):
>
>
>isn't there also the problem that sendfile will be broken on many Linux boxes?
>
>>
>>- Apache/APR compiled with _FILE_OFFSET_BITS=64 has an incompatible
>>module ABI with one which isn't - this is especially nasty if you don't
>>detect it at module load time.
>
>something we can fix for APR 1.0/Apache 2.1-dev by making apr_off_t always 64 bits
+1
>whether native apr_off_t is 64 bits or not?
s/apr_off_t/off_t/ :-) Yes - I like this proposal if the platform has *any*
expectation of being compiled for LARGEFILES. Of course, if you can name
a platform with 32 bit files, that's another story ;-)
Enough users consider this a bug that it might be worth considering some
way of installing both libapr.0.9.so and libapr.lf.0.9.so. I'm not just speaking
of this one case, but all of the problems sending >2GB/4GB content, huge
log files, etc.
Just a thought.
Bill
Bill
Re: apr_lstat() fails on linux for large files
Posted by "William A. Rowe, Jr." <wr...@rowe-clan.net>.
At 03:37 PM 2/28/2003, Jeff Trawick wrote:
>Joe Orton wrote:
>
>>On Fri, Feb 28, 2003 at 02:23:31PM -0500, Cliff Woolley wrote:
>>...
>>
>>>Of course, if I go and explictly compile Apache with
>>>CFLAGS=-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64, then it works.
>>>
>>>So I guess the ultimate questions are these:
>>>
>>>(a) Why doesn't Apache automatically include largefile support?
>>>Performance reasons? Shouldn't there at least be an easy ./configure
>>>option to turn it on?
>>
>>
>>There are at least two ramifications of doing this (this is all only
>>relevant on 32-bit systems):
>
>
>isn't there also the problem that sendfile will be broken on many Linux boxes?
>
>>
>>- Apache/APR compiled with _FILE_OFFSET_BITS=64 has an incompatible
>>module ABI with one which isn't - this is especially nasty if you don't
>>detect it at module load time.
>
>something we can fix for APR 1.0/Apache 2.1-dev by making apr_off_t always 64 bits
+1
>whether native apr_off_t is 64 bits or not?
s/apr_off_t/off_t/ :-) Yes - I like this proposal if the platform has *any*
expectation of being compiled for LARGEFILES. Of course, if you can name
a platform with 32 bit files, that's another story ;-)
Enough users consider this a bug that it might be worth considering some
way of installing both libapr.0.9.so and libapr.lf.0.9.so. I'm not just speaking
of this one case, but all of the problems sending >2GB/4GB content, huge
log files, etc.
Just a thought.
Bill
Bill
Re: apr_lstat() fails on linux for large files
Posted by Jeff Trawick <tr...@attglobal.net>.
Joe Orton wrote:
> On Fri, Feb 28, 2003 at 02:23:31PM -0500, Cliff Woolley wrote:
> ...
>
> >Of course, if I go and explictly compile Apache with
> >CFLAGS=-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64, then it works.
> >
> >So I guess the ultimate questions are these:
> >
> >(a) Why doesn't Apache automatically include largefile support?
> >Performance reasons? Shouldn't there at least be an easy ./configure
> >option to turn it on?
>
>
> There are at least two ramifications of doing this (this is all only
> relevant on 32-bit systems):
isn't there also the problem that sendfile will be broken on many Linux
boxes?
>
> - Apache/APR compiled with _FILE_OFFSET_BITS=64 has an incompatible
> module ABI with one which isn't - this is especially nasty if you don't
> detect it at module load time.
something we can fix for APR 1.0/Apache 2.1-dev by making apr_off_t
always 64 bits whether native apr_off_t is 64 bits or not?
Re: apr_lstat() fails on linux for large files
Posted by Joe Orton <jo...@redhat.com>.
On Fri, Feb 28, 2003 at 02:23:31PM -0500, Cliff Woolley wrote:
...
> Of course, if I go and explictly compile Apache with
> CFLAGS=-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64, then it works.
>
> So I guess the ultimate questions are these:
>
> (a) Why doesn't Apache automatically include largefile support?
> Performance reasons? Shouldn't there at least be an easy ./configure
> option to turn it on?
There are at least two ramifications of doing this (this is all only
relevant on 32-bit systems):
- Apache/APR compiled with _FILE_OFFSET_BITS=64 has an incompatible
module ABI with one which isn't - this is especially nasty if you don't
detect it at module load time.
- Apache modules which are hence built with _FILE_OFFSET_BITS=64 may use
third-party libraries, which may expose off_t in their API. If so, you
require that all those third-party libraries are also compiled with
_FILS_OFFSET_BITS=64. (I don't know how much of a problem this will
really be: a quick grep for off_t through /usr/include and
php4/ext/*/*.c says it might not be *so* bad.)
Regards,
joe