RFC: Unix LFS support for 1.0

[Joe Orton <jo...@manyfish.co.uk>]

A concrete proposal... the two requirements are:

1. APR 1.0 should support large files on Unix systems which have a 
32-bit off_t but support the LFS standard
and
2. APR must not export -D_FILE_OFFSET_BITS=64 by default to get a 64-bit
off_t, since this breaks the ABI of any other packages which use off_t
in their API, and is generally a Bad Thing.  But users should be allowed
to build like this by passing the appropriate CPPFLAGS to configure if
they want.

The proposed solution is that APR must use the "transitional" LFS API on
such platforms, define apr_off_t as off64_t; and use lseek64 and friends
throughout.  Some of this work is already done for Netware, but it
appears to be incomplete since it doesn't use readdir{,_r}64, nor struct
stat64; but maybe Netware does things differently.

If anyone has issues with this approach, please speak up soon!

Work in progress patch attached, todo list currently:

- AIX and Solaris have variants for their sendfile interfaces which take
an off64_t in the appropriate place; need to hook these up and check
HP-UX too (### help welcome! :)

- what to do about old Linux 2.2's which have no sendfile64(): disable
sendfile support, or disable LFS support? both change the ABI

- automatically add -D_LARGEFILE64_SOURCE to CPPFLAGS on platforms which
need it.  Linux doesn't because it's implied by -D_GNU_SOURCE, other
platforms do.

Re: RFC: Unix LFS support for 1.0

[Jeff Trawick <tr...@attglobal.net>]

Joe Orton wrote:
> On Mon, Mar 22, 2004 at 09:16:02AM -0500, Jeff Trawick wrote:
> 
>>Joe Orton wrote:
>>
>>>- what to do about old Linux 2.2's which have no sendfile64(): disable
>>>sendfile support, or disable LFS support? both change the ABI
>>
>>from an ABI standpoint (given that there is no 1.0 ABI yet), it seems safer 
>>to turn off sendfile support and turn on lfs support when sendfile and lfs 
>>conflict...   if an app was built without APR_HAS_SENDFILE and the user 
>>upgraded the OS and rebuilt apr such that apr could support both sendfile 
>>and lfs, the old app binary would still work fine with the new apr, since 
>>adding the sendfile capability would not change the interface to existing 
>>capabilities
> 
> 
> I looked through the various dicussions about this previously...
> 
> How about this: for the Linux case APR_HAS_LARGE_FILES &&
> !defined(HAVE_SENDFILE64) && sizeof(off_t) == 4, just define
> apr_socket_sendfile() as a wrapper which returns EINVAL if passed
> (*offset + *len) > INT_MAX, and otherwise handles thunking between the
> off64_t and off_t correctly for <2gb files?

+1

Re: RFC: Unix LFS support for 1.0

[Joe Orton <jo...@manyfish.co.uk>]

On Mon, Mar 22, 2004 at 09:16:02AM -0500, Jeff Trawick wrote:
> Joe Orton wrote:
> >- AIX and Solaris have variants for their sendfile interfaces which take
> >an off64_t in the appropriate place; need to hook these up and check
> >HP-UX too (### help welcome! :)
> 
> AIX: looks to me that the file offset and bytes-to-send fields were always 
> 64-bit, and there is just one send_file() and just one sendfile-parms 
> structure

Excellent! Thanks.

> >- what to do about old Linux 2.2's which have no sendfile64(): disable
> >sendfile support, or disable LFS support? both change the ABI
> 
> from an ABI standpoint (given that there is no 1.0 ABI yet), it seems safer 
> to turn off sendfile support and turn on lfs support when sendfile and lfs 
> conflict...   if an app was built without APR_HAS_SENDFILE and the user 
> upgraded the OS and rebuilt apr such that apr could support both sendfile 
> and lfs, the old app binary would still work fine with the new apr, since 
> adding the sendfile capability would not change the interface to existing 
> capabilities

I looked through the various dicussions about this previously...

How about this: for the Linux case APR_HAS_LARGE_FILES &&
!defined(HAVE_SENDFILE64) && sizeof(off_t) == 4, just define
apr_socket_sendfile() as a wrapper which returns EINVAL if passed
(*offset + *len) > INT_MAX, and otherwise handles thunking between the
off64_t and off_t correctly for <2gb files?

That means no regression since 0.9.x on such platforms (since 0.9.x
can't do anything with >2gb files let alone sendfile() them); and in
general everything else works the same or better, just with yet another
obscure edge case where apr_socket_sendfile() fails (and more ammunition
to get EnableSendfile be Off by default in httpd :).

> whatever the decision is, a --disable-lfs is useful because the decision 
> will
> be wrong for somebody and there may be specific platform problems that need
> an easy work-around

OK, will add, thanks.

joe

Re: RFC: Unix LFS support for 1.0

[Jeff Trawick <tr...@attglobal.net>]

Joe Orton wrote:
> A concrete proposal... the two requirements are:
> 
> 1. APR 1.0 should support large files on Unix systems which have a 
> 32-bit off_t but support the LFS standard
> and
> 2. APR must not export -D_FILE_OFFSET_BITS=64 by default to get a 64-bit
> off_t, since this breaks the ABI of any other packages which use off_t
> in their API, and is generally a Bad Thing.  But users should be allowed
> to build like this by passing the appropriate CPPFLAGS to configure if
> they want.
> 
> The proposed solution is that APR must use the "transitional" LFS API on
> such platforms, define apr_off_t as off64_t; and use lseek64 and friends
> throughout.  Some of this work is already done for Netware, but it
> appears to be incomplete since it doesn't use readdir{,_r}64, nor struct
> stat64; but maybe Netware does things differently.

+1

> - AIX and Solaris have variants for their sendfile interfaces which take
> an off64_t in the appropriate place; need to hook these up and check
> HP-UX too (### help welcome! :)

AIX: looks to me that the file offset and bytes-to-send fields were always 
64-bit, and there is just one send_file() and just one sendfile-parms structure

> - what to do about old Linux 2.2's which have no sendfile64(): disable
> sendfile support, or disable LFS support? both change the ABI

from an ABI standpoint (given that there is no 1.0 ABI yet), it seems safer to 
turn off sendfile support and turn on lfs support when sendfile and lfs 
conflict...   if an app was built without APR_HAS_SENDFILE and the user 
upgraded the OS and rebuilt apr such that apr could support both sendfile and 
lfs, the old app binary would still work fine with the new apr, since adding 
the sendfile capability would not change the interface to existing capabilities

whatever the decision is, a --disable-lfs is useful because the decision will
be wrong for somebody and there may be specific platform problems that need
an easy work-around