You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Chris Frost <ch...@frostnet.net> on 2007/03/27 19:04:28 UTC
Large file support with apr 1.2.8, svn 1.4.3, and linux?
When I try to checkin 6 GBs in a few thousand files, 'svn ci' fails while
"Transmitting file data ....":
svn: Commit failed (details follow):
svn: Can't write to file '/home/frost/repo_photos/db/transactions/9-1.txn/rev': File too large
I believe this means subversion+apr cannot use Large Files.
The repository's file system has 17 GB free when the ci fails.
Oddly, it looks like I have built svn and apr with large file support.
I was hoping someone here might know what I have missed?
I am running on linux with glibc 2.3.2 (debian 3.1, parisc, 32bit kernel).
I built apr and apr-util 1.2.8 and subversion 1.4.3.
Re: Large file support with apr 1.2.8, svn 1.4.3, and linux?
Posted by Peter Samuelson <pe...@p12n.org>.
[Chris Frost]
> I just created a checkin with a 2.6GB db/rev/K file on etch parisc with the
> official svn package; all good!
> (So it looks like it is only the svn 1.4 build I made on sarge parisc that
> did not support large fsfs revision files.)
Great - thanks for following up on this.
Peter
Re: Large file support with apr 1.2.8, svn 1.4.3, and linux?
Posted by Chris Frost <ch...@frostnet.net>.
On Fri, Apr 06, 2007 at 03:34:00PM -0500, Peter Samuelson wrote:
> [Chris Frost]
> > This is on a production server and I am out of town for a few weeks.
> > We do plan to take the machine out of production after sarge comes
> > out; would you be interested in the results at that point? If so, I
> > can probably test and let you know.
>
> Yeah, that would be great. It'll be nice to confirm that our official
> packages don't have this problem.
I just created a checkin with a 2.6GB db/rev/K file on etch parisc with the
official svn package; all good!
(So it looks like it is only the svn 1.4 build I made on sarge parisc that
did not support large fsfs revision files.)
--
Chris Frost | <http://www.frostnet.net/chris/>
-------------+----------------------------------------
PGP: <http://www.frostnet.net/chris/about/pgp_key.txt>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Large file support with apr 1.2.8, svn 1.4.3, and linux?
Posted by Peter Samuelson <pe...@p12n.org>.
[Chris Frost]
> This is on a production server and I am out of town for a few weeks.
> We do plan to take the machine out of production after sarge comes
> out; would you be interested in the results at that point? If so, I
> can probably test and let you know.
Yeah, that would be great. It'll be nice to confirm that our official
packages don't have this problem.
Thanks,
Peter
Re: Large file support with apr 1.2.8, svn 1.4.3, and linux?
Posted by Chris Frost <ch...@frostnet.net>.
On Thu, Mar 29, 2007 at 08:18:38PM -0500, Peter Samuelson wrote:
> Hrrrrm. Thanks for the strace, even though it makes no sense to me.
> Also, I can't reproduce it on a Debian etch system using official
> packages of apr and subversion. And I'm pretty sure we haven't added
> any relevant patches to either apr or subversion (I did some LFS work
> on apr recently, but nothing that would affect this case). On my
> system, strace shows that apr uses O_LARGEFILE for _all_ open calls.
>
> If it is convenient for you to upgrade your system from sarge to etch
> (which will be released Real Soon Now), I'll be interested to know
> whether you still have problems. (Interested in the sense of "I've put
> in a certain amount of effort to ensure that LFS in apr works properly
> in etch".)
This is on a production server and I am out of town for a few weeks.
We do plan to take the machine out of production after sarge comes out;
would you be interested in the results at that point? If so, I can probably
test and let you know.
--
Chris Frost | <http://www.frostnet.net/chris/>
-------------+----------------------------------------
PGP: <http://www.frostnet.net/chris/about/pgp_key.txt>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Large file support with apr 1.2.8, svn 1.4.3, and linux?
Posted by Peter Samuelson <pe...@p12n.org>.
[Chris Frost]
> This is interesting: wc opens use O_LARGEFILE on fds that will read
> or write but the ra open of the rev file does not use O_LARGEFILE.
Hrrrrm. Thanks for the strace, even though it makes no sense to me.
Also, I can't reproduce it on a Debian etch system using official
packages of apr and subversion. And I'm pretty sure we haven't added
any relevant patches to either apr or subversion (I did some LFS work
on apr recently, but nothing that would affect this case). On my
system, strace shows that apr uses O_LARGEFILE for _all_ open calls.
If it is convenient for you to upgrade your system from sarge to etch
(which will be released Real Soon Now), I'll be interested to know
whether you still have problems. (Interested in the sense of "I've put
in a certain amount of effort to ensure that LFS in apr works properly
in etch".)
Peter
Re: Large file support with apr 1.2.8, svn 1.4.3, and linux?
Posted by Chris Frost <ch...@frostnet.net>.
On Thu, Mar 29, 2007 at 12:35:04AM -0500, Peter Samuelson wrote:
>
> [Chris Frost]
> > svn: Commit failed (details follow):
> > svn: Can't write to file '/home/frost/repo_photos/db/transactions/9-1.txn/rev': File too large
>
> That's odd.
>
> > Both apr-util and subversion's config.logs shows they knows to define
> > _LARGEFILE64_SOURCE. subversion's SWIG includes set _FILE_OFFSET_BITS=64,
> > but no one else seems to do so, at least in their configure log.
>
> That's normal: apr doesn't need _FILE_OFFSET_BITS=64 or
> _LARGEFILE_SOURCE, it uses the facilities of _LARGEFILE64_SOURCE
> instead.
>
> Could you run the svn command using strace, and note particularly the
> open() and write() calls involving files in the transactions directory?
> apr _should_ be calling open with the O_LARGEFILE flag, but who knows.
This is interesting: wc opens use O_LARGEFILE on fds that will read
or write but the ra open of the rev file does not use O_LARGEFILE.
$ grep 'open.*rev"' ci.strace
open("/home/frost/repo_test/db/transactions/0-1.txn/rev", O_WRONLY|O_CREAT|O_EXCL, 0666) = 3
open("/home/frost/repo_test/db/transactions/0-1.txn/rev", O_WRONLY) = 4
open("/home/frost/repo_test/db/transactions/0-1.txn/rev", O_WRONLY) = 5
$ grep 'O_LARGEFILE' ci.strace
open("/usr/lib/locale/locale-archive", O_RDONLY|O_LARGEFILE) = 3
open("/home/frost/co_test/.svn/props/zeros.svn-work", O_RDONLY|O_LARGEFILE) = 3
open("/home/frost/co_test/.svn/tmp/props/zeros.svn-work.tmp", O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE, 0666) = 4
open("/home/frost/co_test/zeros", O_RDONLY|O_LARGEFILE) = 3
open("/home/frost/co_test/.svn/tmp/text-base/zeros.svn-base.tmp", O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE, 0666) = 4
open("/home/frost/repo_test/db/transactions/0-1.txn", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 4
An strace for a 10MB file add checkin to a local repository is attached.
--
Chris Frost | <http://www.frostnet.net/chris/>
-------------+----------------------------------------
PGP: <http://www.frostnet.net/chris/about/pgp_key.txt>
Re: Large file support with apr 1.2.8, svn 1.4.3, and linux?
Posted by Peter Samuelson <pe...@p12n.org>.
[Chris Frost]
> svn: Commit failed (details follow):
> svn: Can't write to file '/home/frost/repo_photos/db/transactions/9-1.txn/rev': File too large
That's odd.
> Both apr-util and subversion's config.logs shows they knows to define
> _LARGEFILE64_SOURCE. subversion's SWIG includes set _FILE_OFFSET_BITS=64,
> but no one else seems to do so, at least in their configure log.
That's normal: apr doesn't need _FILE_OFFSET_BITS=64 or
_LARGEFILE_SOURCE, it uses the facilities of _LARGEFILE64_SOURCE
instead.
Could you run the svn command using strace, and note particularly the
open() and write() calls involving files in the transactions directory?
apr _should_ be calling open with the O_LARGEFILE flag, but who knows.