Re: large file support (was: Re: Memory comsumption of svn)

[Justin Erenkrantz <je...@ebuilt.com>]

On Thu, Feb 28, 2002 at 12:00:16PM -0800, Greg Stein wrote:
> [ moved from dev@subversion.tigris.org mailing list ]
> 
> On Thu, Feb 28, 2002 at 09:21:36AM -0800, Justin Erenkrantz wrote:
> >...
> > This shouldn't be too much of a problem because BDB will try to
> > hide that, but APR will *not* handle large files on Linux since
> > they disable sendfile() with 64bit files.  Since APR is
> > network-oriented and we like sendfile(), we don't allow 64-bit
> > file mode in APR on Linux.  And, consequently, we've never enabled
> > 64-bit support on Unix - only Win32 has largefile support in
> > APR.  -- justin
> 
> Couldn't we have large file support in APR for Linux, but just not allow
> sendfile on those files?
> 
> I find it pretty awful to not have large file support in APR, from a basic
> portability library standpoint.

Nope, glibc will stop us from even *compiling* if we have
FILE_OFFSET_BITS=64 and we include sys/sendfile.h.  So, it isn't
something where we can even be selective about our usage of
sendfile().  It's an all-or-nothing deal on Linux (and again,
sys/sendfile.h comes from glibc, so it isn't really Linux's fault,
but glibc's).

At the time, no one seemed interested enough in having this
capability to flesh out the following patch which should be the
autoconf magic required for 64-bit file support on Unix: 

http://www.apache.org/~jerenkrantz/apr_largefile.m4

It worked on Linux and Solaris here, but since it stopped sendfile()
usage on Linux, I stopped and asked the list what to do.  No one
replied or commented.  =(

I had/have other things that I can spend my time on where I can help
release 2.0.  The group wasn't interested, so I dropped it.  -- justin

Re: [PATCH] Re: large file support (was: Re: Memory comsumption of svn)

["William A. Rowe, Jr." <wr...@rowe-clan.net>]

At 12:05 PM 3/7/2002, you wrote:
>On 7 Mar 2002, Jeff Trawick wrote:
>
>| Any reason this didn't get committed?
>
>If everyone likes this general framework, I can do research and get more
>OSes that have largefile support added into that patch, besides just
>having Solaris >= 2.6 enabled.
>
>Also, I know (Greg? Bill?)) was saying something about APR_HAS_LARGEFILES
>or something of the sort. A grep through the unix tree doesnt show that
>symbol, but is this something that's really needed? I'm of the opinion
>that largefile support exists below the APR layer, and any OS-specific
>compile flags are saved to apxs in Apache-land... any other thoughts on
>this?

APR_HAS_LARGEFILES tells us that files don't fit in memory (and that
apr_size_t is not equivalent to apr_off_t.)

This affects code in several places, it's critical we note it.

For example, we 'know' we aren't allowed to sendfile() files bigger than
memory allows, so we split sendfiles into units (16MB each, in apache)
for transmission.  mmap and other users will have similar concerns.

Bill

Re: [PATCH] Re: large file support (was: Re: Memory comsumption of svn)

[Dale Ghent <da...@elemental.org>]

On Thu, 7 Mar 2002, Justin Erenkrantz wrote:

| I'd rather see us use this approach rather than hardcode default
| values based on OS.  IIRC, it detected the right stuff on Linux
| and Solaris fine.  But, we also need to disable largefiles no
| matter what on Linux (and perhaps other OSes).
|
| I won't have the time to pursue this in the forseeable future,
| so feel free to take this patch and run.  =)  -- justin

Okay! Sounds good to me. Once I get back to Baltimore from Boston where
I've been this week, I'll work on this.

/dale

Re: [PATCH] Re: large file support (was: Re: Memory comsumption of svn)

[Justin Erenkrantz <je...@ebuilt.com>]

On Thu, Mar 07, 2002 at 07:10:50AM -0500, Jeff Trawick wrote:
> Any reason this didn't get committed?

I believe a more generic and proper patch is the one listed at
the beginning of this thread:

http://www.apache.org/~jerenkrantz/apr_largefile.m4

I'd rather see us use this approach rather than hardcode default
values based on OS.  IIRC, it detected the right stuff on Linux
and Solaris fine.  But, we also need to disable largefiles no
matter what on Linux (and perhaps other OSes).  

I won't have the time to pursue this in the forseeable future,
so feel free to take this patch and run.  =)  -- justin

Re: [PATCH] Re: large file support (was: Re: Memory comsumption of svn)

[Dale Ghent <da...@elemental.org>]

On 7 Mar 2002, Jeff Trawick wrote:

| Any reason this didn't get committed?

If everyone likes this general framework, I can do research and get more
OSes that have largefile support added into that patch, besides just
having Solaris >= 2.6 enabled.

Also, I know (Greg? Bill?)) was saying something about APR_HAS_LARGEFILES
or something of the sort. A grep through the unix tree doesnt show that
symbol, but is this something that's really needed? I'm of the opinion
that largefile support exists below the APR layer, and any OS-specific
compile flags are saved to apxs in Apache-land... any other thoughts on
this?

/dale

Re: [PATCH] Re: large file support (was: Re: Memory comsumption of svn)

[Jeff Trawick <tr...@attglobal.net>]

Dale Ghent <da...@elemental.org> writes:

> Just thought I'd quip up and show what I've had working for me re:
> largefiles on Solaris with sendfilev. See the patch attached to this mail.
> 
> Everything seems to work fine. I can serve > 2GB files and write logs past
> the 2GB mark as well. Running 'nm' on libapr.so and httpd show all the
> necessary functions have been redefined ( open() -> open64(), stat()
> ->stat64(), mmap() -> mmap64(), etc )
> 
> /dale
> 
> Index: apr/configure.in

Any reason this didn't get committed?

-- 
Jeff Trawick | trawick@attglobal.net | PGP public key at web site:
       http://www.geocities.com/SiliconValley/Park/9289/
             Born in Roswell... married an alien...

[PATCH] Re: large file support (was: Re: Memory comsumption of svn)

[Dale Ghent <da...@elemental.org>]

Just thought I'd quip up and show what I've had working for me re:
largefiles on Solaris with sendfilev. See the patch attached to this mail.

Everything seems to work fine. I can serve > 2GB files and write logs past
the 2GB mark as well. Running 'nm' on libapr.so and httpd show all the
necessary functions have been redefined ( open() -> open64(), stat()
->stat64(), mmap() -> mmap64(), etc )

/dale