You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Andy Whitcroft <ap...@shadowen.org> on 2003/12/05 19:51:09 UTC
svnadmin hotcopy - fails for very large repositories
Talking to egagnon on IRC it appears that there is an issue with the
svnadmin hotcopy command when the repository is _very_ large, greater than
2Gb. In short it appears that we use apr_open_file to open each file for
copy. This open does not request O_LARGEFILE and so the open fails for
large string database.
In the example egagnon was trying to copy a repository with a strings file
of 3217223680. He was trying to copy it to the same filesystem prior to
backup. The copy itself failed as below:
# svnadmin hotcopy --clean-logs /hd/b/SUBVERSION/
/hd/b/hot-backup-subversion/
svn: File too large
svn: svn_io_copy_file: error copying '/hd/b/SUBVERSION/db/strings' to
'/hd/b/hot-backup-subversion/db/strings.tmp'
#
stracing the command shows the failure:
chmod("/hd/b/hot-backup-subversion/db/representations.tmp", 0644) = 0
rename("/hd/b/hot-backup-subversion/db/representations.tmp",
"/hd/b/hot-backup-subversion/db/representations") = 0
open("/hd/b/hot-backup-subversion/db/strings.tmp",
O_RDWR|O_CREAT|O_EXCL, 0666) = 6
close(6) = 0
open("/hd/b/SUBVERSION/db/strings", O_RDONLY) = -1 EFBIG (File too
large)
write(2, "svn: File too large\n", 20svn: File too large
Testing on this system shows that O_LARGEFILE support is available and used.
open("/hd/b/SUBVERSION/db/strings", O_RDONLY|O_LARGEFILE) = 3
That brings me to the real question. Clearly there is an issue here with
large
repositories. However, it is not clear what is a fault here. Is this an
underlying failure with APR, ie the problem is with the APR library and we
should be looking to filing an issue against that. Or should we be looking
for another way of handling these copies? Coding them ourselves etc?
I have had a brief look at the dev list archives and the issue tracker and
can't find anything relating to this?
Comments or thoughts?
Cheers.
-apw
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: svnadmin hotcopy - fails for very large repositories
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
--On Friday, December 5, 2003 3:28 PM -0500 mark benedetto king
<mb...@lowlatency.com> wrote:
> Right, but it doesn't expose an APR_LARGEFILE flag (analogous to APR_READ).
> Without one, the application can't decide.
>
> apr_file_copy() would probably always use APR_LARGEFILE, if it existed.
FWIW, largefile is a big giant mess that no one wants to implement for APR.
I've submitted several patches to enable it that work on most non brain-dead
OSes. However, the problem with largefile mainly comes from Linux brokenness.
Older versions of the kernel (IIRC, 2.2/2.4) combined with older glibc's
#error out when mixing sendfile and largefile. So, it's been on no one's
priority list to implement. I think 2.6 with current glibc versions will be
the first to allow largefile and sendfile in the same application. -- justin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: svnadmin hotcopy - fails for very large repositories
Posted by mark benedetto king <mb...@lowlatency.com>.
On Fri, Dec 05, 2003 at 08:17:49PM +0000, Andy Whitcroft wrote:
> --On 05 December 2003 15:06 -0500 mark benedetto king <mb...@lowlatency.com>
> wrote:
>
> >APR.
> >
> >apr_file_open() does not turn on O_LARGEFILE.
>
> Without doubt APR doesn't specify O_LARGEFILE. However, it is not always
> possible to just specify it. If we could it would simply be specified all
> the time by libc or indeed in the kernel. It changes 'size' of a number of
> the related objects such as the seek pointer and that needs to be
> considered. I guess what I am saying is that APR is to blame if and only
Right, but it doesn't expose an APR_LARGEFILE flag (analogous to APR_READ).
Without one, the application can't decide.
apr_file_copy() would probably always use APR_LARGEFILE, if it existed.
--ben
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: svnadmin hotcopy - fails for very large repositories
Posted by Andy Whitcroft <ap...@shadowen.org>.
--On 05 December 2003 15:06 -0500 mark benedetto king <mb...@lowlatency.com>
wrote:
> APR.
>
> apr_file_open() does not turn on O_LARGEFILE.
Without doubt APR doesn't specify O_LARGEFILE. However, it is not always
possible to just specify it. If we could it would simply be specified all
the time by libc or indeed in the kernel. It changes 'size' of a number of
the related objects such as the seek pointer and that needs to be
considered. I guess what I am saying is that APR is to blame if and only
if they intend to support opening large files. If APR has this as a
limitation then the 'bug' is in our expectations of it and we can't use it.
Cheers.
-apw
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: svnadmin hotcopy - fails for very large repositories
Posted by mark benedetto king <mb...@lowlatency.com>.
On Fri, Dec 05, 2003 at 07:51:09PM +0000, Andy Whitcroft wrote:
> open("/hd/b/SUBVERSION/db/strings", O_RDONLY) = -1 EFBIG (File too
> large)
> write(2, "svn: File too large\n", 20svn: File too large
>
[...]
> repositories. However, it is not clear what is a fault here. Is this an
> underlying failure with APR, ie the problem is with the APR library and we
> should be looking to filing an issue against that. Or should we be looking
> for another way of handling these copies? Coding them ourselves etc?
>
APR.
apr_file_open() does not turn on O_LARGEFILE.
--ben
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org