You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apr.apache.org by Greg Hudson <gh...@MIT.EDU> on 2004/01/23 00:10:18 UTC
Solving the off_t problem in APR 1.0
I'm going to propose fixing apr_off_t at 64 bits regardless of the
system off_t. This is not about getting large-file support in APR,
although that could come as a side-effect; this is about helping
APR-using libraries to be compatible with two different universes of
libraries, and making APR itself compatible with those two universes.
I'll start at the beginning. 32-bit operating systems wanted to add
large-file support. The *BSDs did this by accepting a little bit of
pain, revving their libc ABI major versions, and winding up with off_t
being 64 bits. Other platforms, notably Linux/x86, did not want to
accept this pain, so they did two things:
* They introduced an explicit API for 64-bit offset support, with a
new offset type (loff_t) and new functions (llseek) to use it.
* They introduced magic compiler flags (-D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64) which cause off_t itself to become 64
bits. Code compiled with these magic flags is ABI-incompatible
with normal code, but it means you can compile old, standard,
portable code unchanged and get large file support as long as you
don't care about ABI issues.
The details may vary once you get into X/Open's involvement, but the
general idea is the same. I think, though I can't verify, that the
idea is that packages would be changed to use the explicit API (when
available), while end builders could use the magic compiler flags as a
crutch in the meantime.
Okay, now enter Perl. Perl, which has always had a certain myopia
when it comes to ABI issues, compiles itself with the magic 64-bit
off_t flags on the relevant platforms. As a consequence, any code
which interfaces to perl (either as bindings or by embedding a perl
interpreter) potentially also needs to compile itself with those
flags, and any code which interfaces to that code (as caller or
callee), and so on. It was a mistake, but not a mistake that Perl can
easily undo. Possibly other packages have made the same error. There
are two non-empty universes of libraries and it hurts when they
collide.
How can we minimize the fallout from these two colliding universes?
By not using off_t in library APIs. If an API does not make use of
off_t, then it is ABI-compatible with both universes regardless of how
its implementation is compiled.
By fixing apr_off_t at 64 bits, we make APR and all APR-using
libraries compatible with both universes. APR could get itself
large-file support by using the explicit 64-bit API under the cover,
or by compiling itself with the large-file support flags, or it could
just not bother; the point here is not to get large file support, but
to be more compatible.
Now, the down sides:
* APR 1.0 would be incompatible with APR 0.9. If any httpd APIs use
apr_off_t (and I'm pretty sure that some do), then httpd 2.x
becomes stuck at APR 0.9. Similarly, if any svn 1.0 APIs use
apr_off_t (currently a matter of debate), then svn 1.x becomes
stuck at APR 0.9. (Subversion developers who are reading this
should take note: regardless of whether APR accepts my proposal,
svn 1.x will experience less pain if it avoids using apr_off_t in
its API. The redundant svn_offset_t type would be much less of a
headache than being stuck at APR 0.9.)
* If, in the future, people find it necessary to break the
9-quintillion-byte barrier and desire 128-bit file offsets, APR
may find itself in an uncomfortable situation by having fixed
apr_off_t at 64 bits. (Though, perhaps no more uncomfortable than
it will in the default situation of using the platform's off_t.)
Re: Solving the off_t problem in APR 1.0
Posted by Kean Johnston <jk...@sco.com>.
> That's asking for a level of ABI guarantee which I don't think APR can
> provide regardless of this apr_off_t issue.
But I agree with Greg, you can address the ABI issues as you find them.
This is one that is easily addressed by APR.
> Will a libapr-0.so built on RHL9 have a compatible ABI with a
> libapr-0.so built on RHL6.2? What if the libpthread soname changed
> between OS versions? (bearing in mind that both applications using APR
> and libapr itself are-or-should-be linked against libpthread)
That is not at all an issue. If app foobar is linked against
libpthread.so.1, and so is APR, then changes to libpthread that result
in libpthread.so.2 wont affect either APR or foobar. The system
integrator / distrubution vendor just needs to make sure that they
provide both libpthread.so.1 and .2. The only time you will get into
trouble is if you recompile APR so that it links against
libpthread.so.2, and the SONAME for APR hasn't changed. In that case
you will have foobar linked to libapr.so.1, which links to
libpthread.so.2, but foorbar is still linked to libpthread.so.1. Thats
just one of those cases where people need to be careful how they
compile vital system libraries.
I dont think that problems as described above should prohibit us
from addressing the off_t issue though.
Kean
Re: Solving the off_t problem in APR 1.0
Posted by Greg Hudson <gh...@MIT.EDU>.
On Wed, 2004-01-28 at 13:00, Kean Johnston wrote:
> > That looks great: I'll commit your patch to the 0.9 branch unless there
> > are any objections. Thanks a lot for working on this issue.
> My personal opinion is that its approaching the issue the wrong way. I
> think first and foremost, we need to establish why ABI compatibility
> with a 0.9 release is so important.
Even if APR adds large-file support in 1.0 by bashing the apr_off_t type
to off64_t where available, it still makes sense to solve this problem
on the 0.9 branch, since httpd 2,x is stuck there and Subversion 1.x
would wind up stuck there.
> On a typical system that we are likely to care about today ... how many
> things are actually using APR that cant be (and arent being) easily
> updated? The two largest consumers of APR are Apache and SubVersion
> right? SVN is chnaging daily, and Apache frequently enough that it is
> unlikely to cause too much pain.
SVN is in a final four-week soak period before its 1.0 release. A
change to the apr_off_t type would cause an incompatible change in SVN's
ABI, which we cannot tolerate before our 2.0 release, which might be
five years off. (It turns out that eliminating apr_off_t from the svn
API is too hard, because of the apr_finfo_t type and because of
scheduling.)
Re: Solving the off_t problem in APR 1.0
Posted by Ben Reser <be...@reser.org>.
On Wed, Jan 28, 2004 at 10:00:30AM -0800, Kean Johnston wrote:
> On a typical system that we are likely to care about today ... how many
> things are actually using APR that cant be (and arent being) easily
> updated? The two largest consumers of APR are Apache and SubVersion
> right? SVN is chnaging daily, and Apache frequently enough that it is
> unlikely to cause too much pain.
SVN is doing a 1.0 release in a month and we're in a freeze except for
critical issues. We have apr_off_t's in our API. We have run into
situations where our ABI breaks because of this. We have two choices,
get this fixed in APR with a nice neat relatively simple fix. Or start
making big ugly hacks in SVN to work around the problem. Nobody really
likes the big ugly hacks that have been proposed.
This problem is indeed causing some pain for SVN and is indeed difficult
to fix in SVN directly. This solution is simply the best solution to
the problem.
> But the question at hand ... as mentioned in the topic ... is what to do
> with 1.0. If ABI compatibility in the 0.9 series is an absolute must,
> the patch proposed will do, I guess. But for 1.0 I really think that
> off_t should be 64 bits (or sensitive to _FILE_OFFSET_BITS - see below),
> and APR does the required juggling inside itself to deal with non-LFS
> system issues. Non-LFS systems are actually the easiest to deal with as
> the largest file size and offset can always be expressed in a 32-bit
> quantity to promoting the internal library type to 64 bits has no impact
> on such systems.
Having coded an attempt to do just that in SVN I don't think it's a
simple as it sounds. You can't feed numbers into a 32-bit system call
that won't fit in a 32-bit type. If you do then you'll have all kinds
of odd behavior. Which means you have to bounds check each and
everytime you make one of those calls. It's not as easy to bounds check
as you one would think.
> One last word on this whole issue. I think it is reasonable for a
> developer to expect to be able to use LFS as and when it needs to. Just
> as users dont have to select a differnt libc when using LFS, they
> shouldn't have to use a different APR. And APR should be suffuciently
> size-agnostic that a developer can use it in one application without LFS
> support and in another with it, with only, at worst, a -D option. It
> seems like people have some kind of allergy to the _FILE_OFFSET_BITS
> approach. I dont understand why. Its a perfectly reasonable way to
> approach the issue. If APR was sensitive to such a macro, then any
> fungling with 64-bit to 32-bit types is unnecessary. For all functions
> in the library that accept or return an apr_off_t, you have _32 and _64
> versions of them. Its not that hard really, and its even pretty clean.
> At least doing that provides a consistant interface.
I agree that it would be nice to have such a setup with LFS when you
wanted it without rebuilding. But that's not really the issue at hand
here. Muddling this issue with LFS, I think just serves to confuse the
issue.
The problem is pretty simple. An application has no way of knowing how
big apr_off_t is as far as the APR library is concerned. Which creates
a pretty nasty situation where an app can be expecting a 64-bit value
but APR only provides a 32-bit value.
The proposed patch fixes only that problem. It doesn't make an attempt
to solve LFS issues for APR. That's something much bigger and more
difficult to solve.
--
Ben Reser <be...@reser.org>
http://ben.reser.org
"Conscience is the inner voice which warns us somebody may be looking."
- H.L. Mencken
Re: Solving the off_t problem in APR 1.0
Posted by Kean Johnston <jk...@sco.com>.
> That looks great: I'll commit your patch to the 0.9 branch unless there
> are any objections. Thanks a lot for working on this issue.
My personal opinion is that its approaching the issue the wrong way. I
think first and foremost, we need to establish why ABI compatibility
with a 0.9 release is so important. I understand its desirable, dont get
me wrong, but it *IS* a 0.9 release. I think people that use software
that is undergoing radical change expect a certain amount of pain, and
part of that pain is dealing with ABI changes.
On a typical system that we are likely to care about today ... how many
things are actually using APR that cant be (and arent being) easily
updated? The two largest consumers of APR are Apache and SubVersion
right? SVN is chnaging daily, and Apache frequently enough that it is
unlikely to cause too much pain.
But the question at hand ... as mentioned in the topic ... is what to do
with 1.0. If ABI compatibility in the 0.9 series is an absolute must,
the patch proposed will do, I guess. But for 1.0 I really think that
off_t should be 64 bits (or sensitive to _FILE_OFFSET_BITS - see below),
and APR does the required juggling inside itself to deal with non-LFS
system issues. Non-LFS systems are actually the easiest to deal with as
the largest file size and offset can always be expressed in a 32-bit
quantity to promoting the internal library type to 64 bits has no impact
on such systems.
One last word on this whole issue. I think it is reasonable for a
developer to expect to be able to use LFS as and when it needs to. Just
as users dont have to select a differnt libc when using LFS, they
shouldn't have to use a different APR. And APR should be suffuciently
size-agnostic that a developer can use it in one application without LFS
support and in another with it, with only, at worst, a -D option. It
seems like people have some kind of allergy to the _FILE_OFFSET_BITS
approach. I dont understand why. Its a perfectly reasonable way to
approach the issue. If APR was sensitive to such a macro, then any
fungling with 64-bit to 32-bit types is unnecessary. For all functions
in the library that accept or return an apr_off_t, you have _32 and _64
versions of them. Its not that hard really, and its even pretty clean.
At least doing that provides a consistant interface.
Kean
Re: Solving the off_t problem in APR 1.0
Posted by Joe Orton <jo...@redhat.com>.
On Tue, Jan 27, 2004 at 07:33:25PM -0800, Ben Reser wrote:
> On Tue, Jan 27, 2004 at 10:32:20AM -0500, Greg Hudson wrote:
> > There is a new idea afloat, incidentally: rather than fix apr_off_t at
> > 64 bits, fix apr_off_t at the size off_t has at configuration time.
> > (There appears to be no off32_t, so that would have to be apr_int32_t on
> > most 32-bit platforms.) That doesn't get large-file support, but it
> > does make apr_off_t independent of _FILE_OFFSET_BITS, and it doesn't
> > cause an ABI change.
>
> Attached is an implementation of this idea. The idea was originally
> suggested by Tobias Ringström in:
> http://marc.theaimsgroup.com/?l=subversion-dev&m=107490548803069&w=2
>
> This also removes some cruft that appears to have been added because
> sizeof(int) == sizeof(long) on some platforms. Which this patch happily
> avoids altogether for us. :)
That looks great: I'll commit your patch to the 0.9 branch unless there
are any objections. Thanks a lot for working on this issue.
joe
Re: Solving the off_t problem in APR 1.0
Posted by "William A. Rowe, Jr." <wr...@rowe-clan.net>.
At 12:48 PM 2/18/2004, Ben Reser wrote:
>On Tue, Feb 17, 2004 at 01:15:12PM -0600, William A. Rowe, Jr. wrote:
>> I am against changing the default size or alignment of any data type
>> in
>> APR_0_9_BRANCH. If this [has] happened we break all binary compat.
>
>Umm we've been careful to avoid doing exactly that. The point here
>isn't to change size of the type. It's to fix it at compile time rather
>than at build time. The last patch sent most definitely does that
>without altering size of anytime and additionally without altering the
>type which would change the name mangling in C++.
Excellent - I will review on a few different unix platforms w.r.t. apr 9.4
v.s. head.
>> Good fix for APR 1.0 along with all the other problematic design
>> decisions
>> we made in 0.9's twisty and winding road. +1 for 1.0
>
>There are potentially cleaner ways to fix this in 1.0. This is a
>temporary work around for a problem.
Ack.
Re: Solving the off_t problem in APR 1.0
Posted by Ben Reser <be...@reser.org>.
On Tue, Feb 17, 2004 at 01:15:12PM -0600, William A. Rowe, Jr. wrote:
> I am against changing the default size or alignment of any data type
> in
> APR_0_9_BRANCH. If this [has] happened we break all binary compat.
Umm we've been careful to avoid doing exactly that. The point here
isn't to change size of the type. It's to fix it at compile time rather
than at build time. The last patch sent most definitely does that
without altering size of anytime and additionally without altering the
type which would change the name mangling in C++.
> Good fix for APR 1.0 along with all the other problematic design
> decisions
> we made in 0.9's twisty and winding road. +1 for 1.0
There are potentially cleaner ways to fix this in 1.0. This is a
temporary work around for a problem.
--
Ben Reser <be...@reser.org>
http://ben.reser.org
"Conscience is the inner voice which warns us somebody may be looking."
- H.L. Mencken
Re: Solving the off_t problem in APR 1.0
Posted by "William A. Rowe, Jr." <wr...@rowe-clan.net>.
At 03:31 AM 2/17/2004, Ben Reser wrote:
>On Wed, Feb 04, 2004 at 03:22:17PM +0000, Joe Orton wrote:
>> On Mon, Feb 02, 2004 at 05:19:09PM -0800, Ben Reser wrote:
>> > On Wed, Jan 28, 2004 at 04:45:26PM -0800, Ben Reser wrote:
>> > > So I guess I'll look into redoing it to use int, long or long long
>> > > instead.
>> >
>> > I found some time to look at this. The types I'm using now match the
>> > formats we were using before. So we shouldn't have an ABI conflict. If
>> > we do we had a bug with the formats already.
>> >
>> > The one case where this may exist would be 64-bit archs with 64-bit
>> > off_t's. These platforms could use long long or long for the off_t.
>> > They might choose differently than we have for apr_int64_t. I don't
>> > know of any other way to deal with this than what was already done with
>> > the LFS platforms that use long for off_t. We'll simply have to detect
>> > these platforms one by one and apply exceptions for them.
>>
>> Alternatively, apr_off_t could be set to an integer type only on
>> platforms where sizeof(int)==4: for real LP64 platforms (and those
>> sizeof(int)==2 platforms which APR really doesn't build on anyway), just
>> leave apr_off_t defined to off_t.
>>
>> This would be perhaps be simpler...
>
>I'm going to try and find time to look at this alternative solution
>tomorrow. I want to have this "closed" in APR before we put out
>Subversion 1.0.
I am against changing the default size or alignment of any data type in
APR_0_9_BRANCH. If this [has] happened we break all binary compat.
Good fix for APR 1.0 along with all the other problematic design decisions
we made in 0.9's twisty and winding road. +1 for 1.0
Bill
Re: Solving the off_t problem in APR 1.0
Posted by Ben Reser <be...@reser.org>.
On Wed, Feb 04, 2004 at 03:22:17PM +0000, Joe Orton wrote:
> On Mon, Feb 02, 2004 at 05:19:09PM -0800, Ben Reser wrote:
> > On Wed, Jan 28, 2004 at 04:45:26PM -0800, Ben Reser wrote:
> > > So I guess I'll look into redoing it to use int, long or long long
> > > instead.
> >
> > I found some time to look at this. The types I'm using now match the
> > formats we were using before. So we shouldn't have an ABI conflict. If
> > we do we had a bug with the formats already.
> >
> > The one case where this may exist would be 64-bit archs with 64-bit
> > off_t's. These platforms could use long long or long for the off_t.
> > They might choose differently than we have for apr_int64_t. I don't
> > know of any other way to deal with this than what was already done with
> > the LFS platforms that use long for off_t. We'll simply have to detect
> > these platforms one by one and apply exceptions for them.
>
> Alternatively, apr_off_t could be set to an integer type only on
> platforms where sizeof(int)==4: for real LP64 platforms (and those
> sizeof(int)==2 platforms which APR really doesn't build on anyway), just
> leave apr_off_t defined to off_t.
>
> This would be perhaps be simpler...
I'm going to try and find time to look at this alternative solution
tomorrow. I want to have this "closed" in APR before we put out
Subversion 1.0.
--
Ben Reser <be...@reser.org>
http://ben.reser.org
"Conscience is the inner voice which warns us somebody may be looking."
- H.L. Mencken
Re: Solving the off_t problem in APR 1.0
Posted by Joe Orton <jo...@manyfish.co.uk>.
On Mon, Feb 02, 2004 at 05:19:09PM -0800, Ben Reser wrote:
> On Wed, Jan 28, 2004 at 04:45:26PM -0800, Ben Reser wrote:
> > So I guess I'll look into redoing it to use int, long or long long
> > instead.
>
> I found some time to look at this. The types I'm using now match the
> formats we were using before. So we shouldn't have an ABI conflict. If
> we do we had a bug with the formats already.
>
> The one case where this may exist would be 64-bit archs with 64-bit
> off_t's. These platforms could use long long or long for the off_t.
> They might choose differently than we have for apr_int64_t. I don't
> know of any other way to deal with this than what was already done with
> the LFS platforms that use long for off_t. We'll simply have to detect
> these platforms one by one and apply exceptions for them.
Alternatively, apr_off_t could be set to an integer type only on
platforms where sizeof(int)==4: for real LP64 platforms (and those
sizeof(int)==2 platforms which APR really doesn't build on anyway), just
leave apr_off_t defined to off_t.
This would be perhaps be simpler...
joe
Re: Solving the off_t problem in APR 1.0
Posted by Ben Reser <be...@reser.org>.
On Tue, Feb 03, 2004 at 04:40:38PM +0000, Joe Orton wrote:
> Thanks - can you make a patch relative to the APR_0_9_BRANCH branch?
Sure, here you go.
--
Ben Reser <be...@reser.org>
http://ben.reser.org
"Conscience is the inner voice which warns us somebody may be looking."
- H.L. Mencken
Re: Solving the off_t problem in APR 1.0
Posted by Joe Orton <jo...@redhat.com>.
On Mon, Feb 02, 2004 at 05:19:09PM -0800, Ben Reser wrote:
> On Wed, Jan 28, 2004 at 04:45:26PM -0800, Ben Reser wrote:
> > So I guess I'll look into redoing it to use int, long or long long
> > instead.
>
> I found some time to look at this. The types I'm using now match the
> formats we were using before. So we shouldn't have an ABI conflict. If
> we do we had a bug with the formats already.
Thanks - can you make a patch relative to the APR_0_9_BRANCH branch?
joe
Re: Solving the off_t problem in APR 1.0
Posted by Ben Reser <be...@reser.org>.
On Wed, Jan 28, 2004 at 04:45:26PM -0800, Ben Reser wrote:
> So I guess I'll look into redoing it to use int, long or long long
> instead.
I found some time to look at this. The types I'm using now match the
formats we were using before. So we shouldn't have an ABI conflict. If
we do we had a bug with the formats already.
The one case where this may exist would be 64-bit archs with 64-bit
off_t's. These platforms could use long long or long for the off_t.
They might choose differently than we have for apr_int64_t. I don't
know of any other way to deal with this than what was already done with
the LFS platforms that use long for off_t. We'll simply have to detect
these platforms one by one and apply exceptions for them.
--
Ben Reser <be...@reser.org>
http://ben.reser.org
"Conscience is the inner voice which warns us somebody may be looking."
- H.L. Mencken
Re: Solving the off_t problem in APR 1.0
Posted by Ben Reser <be...@reser.org>.
On Wed, Jan 28, 2004 at 10:34:41PM +0000, Joe Orton wrote:
> One possible minor issue with this patch:
>
> if a C++ library based on APR uses apr_off_t in its API, I believe that
> changing apr_off_t from a long to an int changes the ABI of that
> library, because of the name mangling stuff.
>
> I don't know of any C++ libraries based on APR, and fixing the
> incompatibility with Perl is probably far more important anyway. Ignore
> it?
D'oh, I'm tempted to say ignore it. But I know rapidsvn is using C++ so
this would probably break rapidsvn if you upgrade APR/httpd2 with this
change in the future. Which I think would be kinda bad.
> We could fudge apr_off_t to still be a long on platforms where it was
> already or even on all platforms with a 32-bit long if anyone really
> cares. It might be wise anyway... could compilers specify different
> structure padding/alignment/calling conventions for int and long even
> though they are the same size? Hmmm...
I doubt the compiler treats int or long any different on the platforms
where they're the same size. I see people misuse int in place of long
and it works fine until they port it to another platform, where it isn't
the same size. This is a big reason why moving code over to 64-bit
archs has been such a hassle.
So I guess I'll look into redoing it to use int, long or long long
instead.
--
Ben Reser <be...@reser.org>
http://ben.reser.org
"Conscience is the inner voice which warns us somebody may be looking."
- H.L. Mencken
Re: Solving the off_t problem in APR 1.0
Posted by Greg Hudson <gh...@MIT.EDU>.
On Thu, 2004-01-29 at 06:32, Joe Orton wrote:
> Not sure what you mean by "APR code"... it would happen if the C++
> library exposes apr_off_t in its own API:
Yes, you're right; Ben corrected me on this over IRC. Sorry to confuse
the issue.
(What I was talking about was taking some or all of the APR
implementation itself and building a C++ library around that, which
presumably no one does.)
Re: Solving the off_t problem in APR 1.0
Posted by Joe Orton <jo...@manyfish.co.uk>.
Ben's fixed-size-apr_off_t patch is now checked in for the next 0.9.x
release, many thanks to all involved.
On Wed, Jan 28, 2004 at 06:37:10PM -0500, Greg Hudson wrote:
> On Wed, 2004-01-28 at 17:34, Joe Orton wrote:
> > One possible minor issue with this patch:
> >
> > if a C++ library based on APR uses apr_off_t in its API, I believe that
> > changing apr_off_t from a long to an int changes the ABI of that
> > library, because of the name mangling stuff.
>
> To clarify, this would only be true of a library which takes APR code
> and compiles it using a C++ compiler, which is a bit of a stretch. C++
> code which merely calls into APR necessarily does so with unmangled
> symbols.
Not sure what you mean by "APR code"... it would happen if the C++
library exposes apr_off_t in its own API:
$ cat apr.cc
#include <sys/types.h>
typedef off_t apr_off_t;
apr_off_t SomeCppFunc(apr_off_t foo) { return foo + 1; }
$ cat apr2.cc
typedef int apr_off_t;
apr_off_t SomeCppFunc(apr_off_t foo) { return foo + 1; }
$ g++ -shared apr.cc
$ nm a.out | grep SomeCpp
000005f0 T _Z11SomeCppFuncl
$ g++ -shared apr2.cc
$ nm a.out | grep SomeCpp
000005f0 T _Z11SomeCppFunci
Re: Solving the off_t problem in APR 1.0
Posted by Greg Hudson <gh...@MIT.EDU>.
On Wed, 2004-01-28 at 17:34, Joe Orton wrote:
> One possible minor issue with this patch:
>
> if a C++ library based on APR uses apr_off_t in its API, I believe that
> changing apr_off_t from a long to an int changes the ABI of that
> library, because of the name mangling stuff.
To clarify, this would only be true of a library which takes APR code
and compiles it using a C++ compiler, which is a bit of a stretch. C++
code which merely calls into APR necessarily does so with unmangled
symbols.
> We could fudge apr_off_t to still be a long on platforms where it was
> already or even on all platforms with a 32-bit long if anyone really
> cares. It might be wise anyway... could compilers specify different
> structure padding/alignment/calling conventions for int and long even
> though they are the same size? Hmmm...
So what if they could? We don't require that apr_off_t and off_t have
the same alignment or calling conventions, only that you can cast
between them without losing information.
(If there are existing platforms, which APR supports, which have this
bizarre property, then the proposed change could affect the 0.9 ABI on
those platforms. But that's even more far-fetched than the idea that we
might run into such platforms in the future.)
Re: Solving the off_t problem in APR 1.0
Posted by Joe Orton <jo...@manyfish.co.uk>.
One possible minor issue with this patch:
if a C++ library based on APR uses apr_off_t in its API, I believe that
changing apr_off_t from a long to an int changes the ABI of that
library, because of the name mangling stuff.
I don't know of any C++ libraries based on APR, and fixing the
incompatibility with Perl is probably far more important anyway. Ignore
it?
We could fudge apr_off_t to still be a long on platforms where it was
already or even on all platforms with a 32-bit long if anyone really
cares. It might be wise anyway... could compilers specify different
structure padding/alignment/calling conventions for int and long even
though they are the same size? Hmmm...
joe
Re: Solving the off_t problem in APR 1.0
Posted by Ben Reser <be...@reser.org>.
On Tue, Jan 27, 2004 at 10:32:20AM -0500, Greg Hudson wrote:
> There is a new idea afloat, incidentally: rather than fix apr_off_t at
> 64 bits, fix apr_off_t at the size off_t has at configuration time.
> (There appears to be no off32_t, so that would have to be apr_int32_t on
> most 32-bit platforms.) That doesn't get large-file support, but it
> does make apr_off_t independent of _FILE_OFFSET_BITS, and it doesn't
> cause an ABI change.
Attached is an implementation of this idea. The idea was originally
suggested by Tobias Ringström in:
http://marc.theaimsgroup.com/?l=subversion-dev&m=107490548803069&w=2
This also removes some cruft that appears to have been added because
sizeof(int) == sizeof(long) on some platforms. Which this patch happily
avoids altogether for us. :)
--
Ben Reser <be...@reser.org>
http://ben.reser.org
"Conscience is the inner voice which warns us somebody may be looking."
- H.L. Mencken
Re: Solving the off_t problem in APR 1.0
Posted by Joe Orton <jo...@redhat.com>.
On Tue, Jan 27, 2004 at 10:32:20AM -0500, Greg Hudson wrote:
> On Tue, 2004-01-27 at 05:49, Joe Orton wrote:
> > > Won't that create an ABI time bomb for platforms which have no
> > > large-file support now, but acquire it in the future?
> >
> > That's asking for a level of ABI guarantee which I don't think APR can
> > provide regardless of this apr_off_t issue.
> >
> > Will a libapr-0.so built on RHL9 have a compatible ABI with a
> > libapr-0.so built on RHL6.2?
>
> Generally, yes. Why wouldn't it? And if it is incompatible, how can we
> ensure that they have different sonames?
I've seen enough interesting things happen with symbol versioning that
I'd not *expect* it to work. APR already has an ABI-breaker across OS
versions in the IPv6 support. I really don't think this is an issue
worth worrying about.
> > What if the libpthread soname changed between OS versions?
>
> Libraries like libc and libpthread very rarely change sonames, precisely
> because it effectively breaks the ABI of every library which depends on
> them. The libc soname change in *BSD for the off_t was extraordinary.
Well, it looks like the libc soname changed in FreeBSD 5, and has
changed regularly in OpenBSD over the last few years. Not *so*
extraordinary on some platforms at least...
> There is a new idea afloat, incidentally: rather than fix apr_off_t at
> 64 bits, fix apr_off_t at the size off_t has at configuration time.
> (There appears to be no off32_t, so that would have to be apr_int32_t on
> most 32-bit platforms.) That doesn't get large-file support, but it
> does make apr_off_t independent of _FILE_OFFSET_BITS, and it doesn't
> cause an ABI change.
Sounds like a good solution for the 0.9 branch at least; note that
APR_OFF_T_FMT would need to change for that though since apr_off_t would
no longer be a long int.
Regards,
joe
Re: Solving the off_t problem in APR 1.0
Posted by Greg Hudson <gh...@MIT.EDU>.
On Tue, 2004-01-27 at 05:49, Joe Orton wrote:
> > Won't that create an ABI time bomb for platforms which have no
> > large-file support now, but acquire it in the future?
>
> That's asking for a level of ABI guarantee which I don't think APR can
> provide regardless of this apr_off_t issue.
>
> Will a libapr-0.so built on RHL9 have a compatible ABI with a
> libapr-0.so built on RHL6.2?
Generally, yes. Why wouldn't it? And if it is incompatible, how can we
ensure that they have different sonames?
> What if the libpthread soname changed between OS versions?
Libraries like libc and libpthread very rarely change sonames, precisely
because it effectively breaks the ABI of every library which depends on
them. The libc soname change in *BSD for the off_t was extraordinary.
There is a new idea afloat, incidentally: rather than fix apr_off_t at
64 bits, fix apr_off_t at the size off_t has at configuration time.
(There appears to be no off32_t, so that would have to be apr_int32_t on
most 32-bit platforms.) That doesn't get large-file support, but it
does make apr_off_t independent of _FILE_OFFSET_BITS, and it doesn't
cause an ABI change.
Re: Solving the off_t problem in APR 1.0
Posted by Joe Orton <jo...@redhat.com>.
On Fri, Jan 23, 2004 at 12:45:43PM -0500, Greg Hudson wrote:
> (Please keep me cc'd on replies, incidentally.)
>
> Joe Orton wrote:
> > I think the best way to achieve this is to define apr_off_t as
> > off64_t on such platforms, rather than unconditionally change
> > apr_off_t everywhere, and add LFS support to APR: most of this work
> > is already done inside #ifdef NETWARE anyway.
>
> Won't that create an ABI time bomb for platforms which have no
> large-file support now, but acquire it in the future?
That's asking for a level of ABI guarantee which I don't think APR can
provide regardless of this apr_off_t issue.
Will a libapr-0.so built on RHL9 have a compatible ABI with a
libapr-0.so built on RHL6.2? What if the libpthread soname changed
between OS versions? (bearing in mind that both applications using APR
and libapr itself are-or-should-be linked against libpthread)
joe
Re: Solving the off_t problem in APR 1.0
Posted by Greg Hudson <gh...@MIT.EDU>.
(Please keep me cc'd on replies, incidentally.)
Joe Orton wrote:
> I think the best way to achieve this is to define apr_off_t as
> off64_t on such platforms, rather than unconditionally change
> apr_off_t everywhere, and add LFS support to APR: most of this work
> is already done inside #ifdef NETWARE anyway.
Won't that create an ABI time bomb for platforms which have no
large-file support now, but acquire it in the future?
> Using interfaces which take a 32-bit off_t from a 64-bit apr_off_t
> sounds like it will be messy.
Yes, but it's internal messiness. Even if we just truncate for now,
that's something we can fix in the future without creating an ABI
issue.
> Why "notably Linux", you keep saying that?
Because Linux is a more mainstream platform, at least from the
viewpoint of APR-using projects, than the others. The point is to
show that this is not just a problem for some relatively obscure
platform.
> The final LFS spec was published in 1996, Solaris implemented it in
> 1997 at least, Linux distributions only started supporting it around
> 2000.
I think Linux implemented its own non-standard API (loff_t, llseek(),
etc.) earlier than that. Not easy for me to check.
> I don't believe fixing Perl is rocket science: it too should just
> use the transitional API, use off64_t for its Off_t, etc.
That may be so; I wasn't aware that perl had its own Off_t type.
Re: Solving the off_t problem in APR 1.0
Posted by Joe Orton <jo...@manyfish.co.uk>.
On Thu, Jan 22, 2004 at 06:10:18PM -0500, Greg Hudson wrote:
> I'm going to propose fixing apr_off_t at 64 bits regardless of the
> system off_t. This is not about getting large-file support in APR,
> although that could come as a side-effect; this is about helping
> APR-using libraries to be compatible with two different universes of
> libraries, and making APR itself compatible with those two universes.
The goal here is really that the size of apr_off_t should not be
affected by the _FILE_OFFSET_BITS definition on 32-bit platforms with
LFS support, which is an admirable goal. I think the best way to
achieve this is to define apr_off_t as off64_t on such platforms, rather
than unconditionally change apr_off_t everywhere, and add LFS support to
APR: most of this work is already done inside #ifdef NETWARE anyway.
Using interfaces which take a 32-bit off_t from a 64-bit apr_off_t
sounds like it will be messy.
> I'll start at the beginning. 32-bit operating systems wanted to add
> large-file support. The *BSDs did this by accepting a little bit of
> pain, revving their libc ABI major versions, and winding up with off_t
> being 64 bits. Other platforms, notably Linux/x86, did not want to
> accept this pain, so they did two things:
Why "notably Linux", you keep saying that? The final LFS spec was
published in 1996, Solaris implemented it in 1997 at least, Linux
distributions only started supporting it around 2000.
...
> Okay, now enter Perl. Perl, which has always had a certain myopia
> when it comes to ABI issues, compiles itself with the magic 64-bit
> off_t flags on the relevant platforms. As a consequence, any code
> which interfaces to perl (either as bindings or by embedding a perl
> interpreter) potentially also needs to compile itself with those
> flags, and any code which interfaces to that code (as caller or
> callee), and so on. It was a mistake, but not a mistake that Perl can
> easily undo. Possibly other packages have made the same error. There
> are two non-empty universes of libraries and it hurts when they
> collide.
I don't believe fixing Perl is rocket science: it too should just use
the transitional API, use off64_t for its Off_t, etc.
Regards,
joe
Re: Solving the off_t problem in APR 1.0
Posted by Garrett Rooney <ro...@electricjellyfish.net>.
On Jan 22, 2004, at 6:10 PM, Greg Hudson wrote:
> Now, the down sides:
>
> * APR 1.0 would be incompatible with APR 0.9. If any httpd APIs use
> apr_off_t (and I'm pretty sure that some do), then httpd 2.x
> becomes stuck at APR 0.9. Similarly, if any svn 1.0 APIs use
> apr_off_t (currently a matter of debate), then svn 1.x becomes
> stuck at APR 0.9. (Subversion developers who are reading this
> should take note: regardless of whether APR accepts my proposal,
> svn 1.x will experience less pain if it avoids using apr_off_t in
> its API. The redundant svn_offset_t type would be much less of a
> headache than being stuck at APR 0.9.)
httpd 2.x is already stuck at APR 0.9 because of the API changes
(deprecated functions being removed and so forth) in APR 1.0.
-garrett