You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Jeremy Conlin <jl...@gmail.com> on 2010/01/19 20:03:23 UTC
svn dump and load not preserving all files
I am migrating my repository to a new server. This requires that I
dump the original and the create a new repository on the new server
and use the svnadmin load command to import everything. I have been
able to do this for a few of my repositories, but one in particular
isn't working. The dump file for this repository is > 9 GB. The load
command starts out, but never finishes. The size of my repository
gets to 2.1 GB and then it seems to stop even though the svnadmin load
command is still running. Eventually (> 24 hours) the command stops
(finishes) but the size of my repository is only 2.1 GB when it should
be around 9 GB. Also, not all of the files have been restored.
Is there a limit in the size of a dump file that can be used? I would
be surprised if that limit were around 9 GB. I'm sure there are
repositories that are much larger than that.
Any ideas on what could be wrong?
Thanks,
Jeremy
Re: svn dump and load not preserving all files
Posted by David Brodbeck <br...@u.washington.edu>.
On Jan 19, 2010, at 12:25 PM, Thomas Loy wrote:
> UNIX has actual physical limits to file size determined by the number of bytes a 32 bit file pointer can index, about 2.4 GB. for older file systems or runtimes.
The OP said it was a Linux server, and it would have to be a *really* old version of Linux to have a 2.4 GB limit. Kernel versions 2.4.x and later support large files, when paired with glibc 2.2.3 or later. By my estimation, you'd have to be running a kernel at least nine years old to run into that issue. I'm not going to say no one is running a system that obsolete, but if they are, they probably know it.
There *are* filesystems that still have that limit under Linux, but none of them are filesystems you're likely to host a repository on. Examples would be VFAT and SMBFS.
--
David Brodbeck
System Administrator, Linguistics
University of Washington
RE: svn dump and load not preserving all files
Posted by Thomas Loy <Th...@cbeyond.net>.
UNIX has actual physical limits to file size determined by the number of bytes a 32 bit file pointer can index, about 2.4 GB. for older file systems or runtimes.
Depending on your system, you may or may not have large file support. Try
man fopen64
if you have an older UNIX.
Cheers,
Tom Loy
-----Original Message-----
From: Jeremy Conlin [mailto:jlconlin@gmail.com]
Sent: Tuesday, January 19, 2010 3:22 PM
To: Thomas Loy
Cc: users@subversion.apache.org
Subject: Re: svn dump and load not preserving all files
On Tue, Jan 19, 2010 at 1:05 PM, Thomas Loy <Th...@cbeyond.net> wrote:
> Which OS? Some operating systems have file size limits of 4 GB or less.
>
> Cheers,
>
> Tom Loy
>
The original OS was on a Mac, the new OS is some *nix server, probably
Linux. I don't believe either of these have limitations as small as
4GB. Am I wrong?
Thanks,
Jeremy
>
> -----Original Message-----
> From: Jeremy Conlin [mailto:jlconlin@gmail.com]
> Sent: Tuesday, January 19, 2010 3:03 PM
> To: users@subversion.apache.org
> Subject: svn dump and load not preserving all files
>
> I am migrating my repository to a new server. This requires that I
> dump the original and the create a new repository on the new server
> and use the svnadmin load command to import everything. I have been
> able to do this for a few of my repositories, but one in particular
> isn't working. The dump file for this repository is > 9 GB. The load
> command starts out, but never finishes. The size of my repository
> gets to 2.1 GB and then it seems to stop even though the svnadmin load
> command is still running. Eventually (> 24 hours) the command stops
> (finishes) but the size of my repository is only 2.1 GB when it should
> be around 9 GB. Also, not all of the files have been restored.
>
> Is there a limit in the size of a dump file that can be used? I would
> be surprised if that limit were around 9 GB. I'm sure there are
> repositories that are much larger than that.
>
> Any ideas on what could be wrong?
>
> Thanks,
> Jeremy
>
Re: svn dump and load not preserving all files
Posted by Jeremy Conlin <jl...@gmail.com>.
On Tue, Jan 19, 2010 at 1:27 PM, Mark Phippard <ma...@gmail.com> wrote:
> On Tue, Jan 19, 2010 at 3:21 PM, Jeremy Conlin <jl...@gmail.com> wrote:
>> On Tue, Jan 19, 2010 at 1:05 PM, Thomas Loy <Th...@cbeyond.net> wrote:
>>> Which OS? Some operating systems have file size limits of 4 GB or less.
>>>
>>> Cheers,
>>>
>>> Tom Loy
>>>
>> The original OS was on a Mac, the new OS is some *nix server, probably
>> Linux. I don't believe either of these have limitations as small as
>> 4GB. Am I wrong?
>
> I doubt any OS has a limit like that. What can have a limit is 32-bit
> integers for file size. Typically if your SVN is compiled with APR
> 1.x then it supports files > 2 GB and if it is compiled with APR 0.9.x
> then it does not.
>
> I do not think this would be your problem anyway. The dump file is
> streamed into svnadmin, so it does not see it as a 9GB file anyway.
> And for the repository itself, it would only manifest as a problem if
> you had a single revision that was 2 GB. Given that it is a dump
> file, perhaps if a single file in the dump file was 2 GB it could be a
> problem since the size would too big for it to process.
>
> Remember the limit is for a single file in the filesystem. For a
> repository, that usually means the revision file, which means you
> would need a revision this big. The entire repository could still
> have a million revision that were 1.9 GB each and be fine.
I do have about 16 files that are each a few hundred MB in size, but
none (that I'm aware of) is over 1GB. All of the large files were
originally committed in a single revision.
Jeremy
Re: svn dump and load not preserving all files
Posted by Mark Phippard <ma...@gmail.com>.
On Tue, Jan 19, 2010 at 3:21 PM, Jeremy Conlin <jl...@gmail.com> wrote:
> On Tue, Jan 19, 2010 at 1:05 PM, Thomas Loy <Th...@cbeyond.net> wrote:
>> Which OS? Some operating systems have file size limits of 4 GB or less.
>>
>> Cheers,
>>
>> Tom Loy
>>
> The original OS was on a Mac, the new OS is some *nix server, probably
> Linux. I don't believe either of these have limitations as small as
> 4GB. Am I wrong?
I doubt any OS has a limit like that. What can have a limit is 32-bit
integers for file size. Typically if your SVN is compiled with APR
1.x then it supports files > 2 GB and if it is compiled with APR 0.9.x
then it does not.
I do not think this would be your problem anyway. The dump file is
streamed into svnadmin, so it does not see it as a 9GB file anyway.
And for the repository itself, it would only manifest as a problem if
you had a single revision that was 2 GB. Given that it is a dump
file, perhaps if a single file in the dump file was 2 GB it could be a
problem since the size would too big for it to process.
Remember the limit is for a single file in the filesystem. For a
repository, that usually means the revision file, which means you
would need a revision this big. The entire repository could still
have a million revision that were 1.9 GB each and be fine.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
RE: svn dump and load not preserving all files
Posted by Thomas Loy <Th...@cbeyond.net>.
UNIX has actual physical limits to file size determined by the number of bytes a 32 bit file pointer can index, about 2.4 GB. for older file systems or runtimes.
Depending on your system, you may or may not have large file support. Try
man fopen64
if you have an older UNIX.
Cheers,
Tom Loy
-----Original Message-----
From: Jeremy Conlin [mailto:jlconlin@gmail.com]
Sent: Tuesday, January 19, 2010 3:22 PM
To: Thomas Loy
Cc: users@subversion.apache.org
Subject: Re: svn dump and load not preserving all files
On Tue, Jan 19, 2010 at 1:05 PM, Thomas Loy <Th...@cbeyond.net> wrote:
> Which OS? Some operating systems have file size limits of 4 GB or less.
>
> Cheers,
>
> Tom Loy
>
The original OS was on a Mac, the new OS is some *nix server, probably
Linux. I don't believe either of these have limitations as small as
4GB. Am I wrong?
Thanks,
Jeremy
>
> -----Original Message-----
> From: Jeremy Conlin [mailto:jlconlin@gmail.com]
> Sent: Tuesday, January 19, 2010 3:03 PM
> To: users@subversion.apache.org
> Subject: svn dump and load not preserving all files
>
> I am migrating my repository to a new server. This requires that I
> dump the original and the create a new repository on the new server
> and use the svnadmin load command to import everything. I have been
> able to do this for a few of my repositories, but one in particular
> isn't working. The dump file for this repository is > 9 GB. The load
> command starts out, but never finishes. The size of my repository
> gets to 2.1 GB and then it seems to stop even though the svnadmin load
> command is still running. Eventually (> 24 hours) the command stops
> (finishes) but the size of my repository is only 2.1 GB when it should
> be around 9 GB. Also, not all of the files have been restored.
>
> Is there a limit in the size of a dump file that can be used? I would
> be surprised if that limit were around 9 GB. I'm sure there are
> repositories that are much larger than that.
>
> Any ideas on what could be wrong?
>
> Thanks,
> Jeremy
>
Re: svn dump and load not preserving all files
Posted by Andy Levy <an...@gmail.com>.
On Tue, Jan 19, 2010 at 15:21, Jeremy Conlin <jl...@gmail.com> wrote:
> On Tue, Jan 19, 2010 at 1:05 PM, Thomas Loy <Th...@cbeyond.net> wrote:
>> Which OS? Some operating systems have file size limits of 4 GB or less.
>>
>> Cheers,
>>
>> Tom Loy
>>
> The original OS was on a Mac, the new OS is some *nix server, probably
> Linux. I don't believe either of these have limitations as small as
> 4GB. Am I wrong?
Depends on what filesystem(s) you're using.
>>
>> -----Original Message-----
>> From: Jeremy Conlin [mailto:jlconlin@gmail.com]
>> Sent: Tuesday, January 19, 2010 3:03 PM
>> To: users@subversion.apache.org
>> Subject: svn dump and load not preserving all files
>>
>> I am migrating my repository to a new server. This requires that I
>> dump the original and the create a new repository on the new server
>> and use the svnadmin load command to import everything. I have been
>> able to do this for a few of my repositories, but one in particular
>> isn't working. The dump file for this repository is > 9 GB. The load
>> command starts out, but never finishes. The size of my repository
>> gets to 2.1 GB and then it seems to stop even though the svnadmin load
>> command is still running. Eventually (> 24 hours) the command stops
>> (finishes) but the size of my repository is only 2.1 GB when it should
>> be around 9 GB. Also, not all of the files have been restored.
>>
>> Is there a limit in the size of a dump file that can be used? I would
>> be surprised if that limit were around 9 GB. I'm sure there are
>> repositories that are much larger than that.
>>
>> Any ideas on what could be wrong?
>>
>> Thanks,
>> Jeremy
>>
>
Re: svn dump and load not preserving all files
Posted by Jeremy Conlin <jl...@gmail.com>.
On Tue, Jan 19, 2010 at 1:05 PM, Thomas Loy <Th...@cbeyond.net> wrote:
> Which OS? Some operating systems have file size limits of 4 GB or less.
>
> Cheers,
>
> Tom Loy
>
The original OS was on a Mac, the new OS is some *nix server, probably
Linux. I don't believe either of these have limitations as small as
4GB. Am I wrong?
Thanks,
Jeremy
>
> -----Original Message-----
> From: Jeremy Conlin [mailto:jlconlin@gmail.com]
> Sent: Tuesday, January 19, 2010 3:03 PM
> To: users@subversion.apache.org
> Subject: svn dump and load not preserving all files
>
> I am migrating my repository to a new server. This requires that I
> dump the original and the create a new repository on the new server
> and use the svnadmin load command to import everything. I have been
> able to do this for a few of my repositories, but one in particular
> isn't working. The dump file for this repository is > 9 GB. The load
> command starts out, but never finishes. The size of my repository
> gets to 2.1 GB and then it seems to stop even though the svnadmin load
> command is still running. Eventually (> 24 hours) the command stops
> (finishes) but the size of my repository is only 2.1 GB when it should
> be around 9 GB. Also, not all of the files have been restored.
>
> Is there a limit in the size of a dump file that can be used? I would
> be surprised if that limit were around 9 GB. I'm sure there are
> repositories that are much larger than that.
>
> Any ideas on what could be wrong?
>
> Thanks,
> Jeremy
>
RE: svn dump and load not preserving all files
Posted by Thomas Loy <Th...@cbeyond.net>.
Which OS? Some operating systems have file size limits of 4 GB or less.
Cheers,
Tom Loy
-----Original Message-----
From: Jeremy Conlin [mailto:jlconlin@gmail.com]
Sent: Tuesday, January 19, 2010 3:03 PM
To: users@subversion.apache.org
Subject: svn dump and load not preserving all files
I am migrating my repository to a new server. This requires that I
dump the original and the create a new repository on the new server
and use the svnadmin load command to import everything. I have been
able to do this for a few of my repositories, but one in particular
isn't working. The dump file for this repository is > 9 GB. The load
command starts out, but never finishes. The size of my repository
gets to 2.1 GB and then it seems to stop even though the svnadmin load
command is still running. Eventually (> 24 hours) the command stops
(finishes) but the size of my repository is only 2.1 GB when it should
be around 9 GB. Also, not all of the files have been restored.
Is there a limit in the size of a dump file that can be used? I would
be surprised if that limit were around 9 GB. I'm sure there are
repositories that are much larger than that.
Any ideas on what could be wrong?
Thanks,
Jeremy
Re: svn dump and load not preserving all files
Posted by Ryan Schmidt <su...@ryandesign.com>.
On Jan 19, 2010, at 15:08, Jeremy Conlin wrote:
> Disregard this message. I just realized that my new server only has
> svn 1.4.2 while my original was using 1.6. I'm sure that is the
> problem. I feel stupid that I didn't check this earlier.
Either Subversion 1.4 or 1.6 can be compiled with either APR 0.9 or APR 1.x. It's probably more the APR version that's the problem, not the version of Subversion.
Re: svn dump and load not preserving all files
Posted by Jeremy Conlin <jl...@gmail.com>.
On Tue, Jan 19, 2010 at 2:18 PM, Mark Phippard <ma...@gmail.com> wrote:
>> Disregard this message. I just realized that my new server only has
>> svn 1.4.2 while my original was using 1.6. I'm sure that is the
>> problem. I feel stupid that I didn't check this earlier.
>
> That should not be a problem in terms of moving a dump file between
> them. 1.4 might not have some features you want, but it should still
> load the file.
I installed svn 1.6.6 in my home directory on the remote server and
used it load the dump file and everything worked without any trouble.
So either the original 1.4.2 on the server was missing some compile
options or it was necessary to use 1.6.6.
Things seem to be looking up now. Thanks everyone for their help.
Jeremy
Re: svn dump and load not preserving all files
Posted by Mark Phippard <ma...@gmail.com>.
> Disregard this message. I just realized that my new server only has
> svn 1.4.2 while my original was using 1.6. I'm sure that is the
> problem. I feel stupid that I didn't check this earlier.
That should not be a problem in terms of moving a dump file between
them. 1.4 might not have some features you want, but it should still
load the file.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
Re: svn dump and load not preserving all files
Posted by Jeremy Conlin <jl...@gmail.com>.
On Tue, Jan 19, 2010 at 2:05 PM, Ryan Schmidt
<su...@ryandesign.com> wrote:
>
> On Jan 19, 2010, at 14:38, Jeremy Conlin wrote:
>
>> On Tue, Jan 19, 2010 at 1:34 PM, Mark Phippard wrote
>>
>>> Basically make
>>> sure you are using SVN that was compiled against httpd 2.2/apr 1.x.
>>
>> Yes this is most likely true. I have about 16 files that are a few
>> hundred MB each and were all originally committed in the same
>> revision.
>>
>> I don't own the server where I'm migrating to. What command can I run
>> to determine what the compile options were?
>
> I'm not aware of a command you could run on the client that would get the server to tell you. You'd have to run e.g. otool on the programs and libraries that are on the server. If you don't have access, ask those who do to do it for you.
>
>
Disregard this message. I just realized that my new server only has
svn 1.4.2 while my original was using 1.6. I'm sure that is the
problem. I feel stupid that I didn't check this earlier.
Thanks,
Jeremy
Re: svn dump and load not preserving all files
Posted by Ryan Schmidt <su...@ryandesign.com>.
On Jan 19, 2010, at 14:38, Jeremy Conlin wrote:
> On Tue, Jan 19, 2010 at 1:34 PM, Mark Phippard wrote
>
>> Basically make
>> sure you are using SVN that was compiled against httpd 2.2/apr 1.x.
>
> Yes this is most likely true. I have about 16 files that are a few
> hundred MB each and were all originally committed in the same
> revision.
>
> I don't own the server where I'm migrating to. What command can I run
> to determine what the compile options were?
I'm not aware of a command you could run on the client that would get the server to tell you. You'd have to run e.g. otool on the programs and libraries that are on the server. If you don't have access, ask those who do to do it for you.
Re: svn dump and load not preserving all files
Posted by Jeremy Conlin <jl...@gmail.com>.
On Tue, Jan 19, 2010 at 1:34 PM, Mark Phippard <ma...@gmail.com> wrote:
> On Tue, Jan 19, 2010 at 3:32 PM, Jeremy Conlin <jl...@gmail.com> wrote:
>> I did redirect the output to a text file. The command never throws an
>> error, but never completes either. There should be 24 revisions in
>> the repository, but when I check out from the new server, there is
>> only 1, so something isn't working right.
>
> OK, definitely.
>
> And if your dump file is 9 GB with only 24 revisions, there is a good
> chance the problem is the size of a single revision. Basically make
> sure you are using SVN that was compiled against httpd 2.2/apr 1.x.
>
Yes this is most likely true. I have about 16 files that are a few
hundred MB each and were all originally committed in the same
revision.
I don't own the server where I'm migrating to. What command can I run
to determine what the compile options were?
Thanks,
Jeremy
Re: svn dump and load not preserving all files
Posted by Mark Phippard <ma...@gmail.com>.
On Tue, Jan 19, 2010 at 3:32 PM, Jeremy Conlin <jl...@gmail.com> wrote:
> I did redirect the output to a text file. The command never throws an
> error, but never completes either. There should be 24 revisions in
> the repository, but when I check out from the new server, there is
> only 1, so something isn't working right.
OK, definitely.
And if your dump file is 9 GB with only 24 revisions, there is a good
chance the problem is the size of a single revision. Basically make
sure you are using SVN that was compiled against httpd 2.2/apr 1.x.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
Re: svn dump and load not preserving all files
Posted by Jeremy Conlin <jl...@gmail.com>.
On Tue, Jan 19, 2010 at 1:05 PM, Mark Phippard <ma...@gmail.com> wrote:
> On Tue, Jan 19, 2010 at 3:03 PM, Jeremy Conlin <jl...@gmail.com> wrote:
>> I am migrating my repository to a new server. This requires that I
>> dump the original and the create a new repository on the new server
>> and use the svnadmin load command to import everything. I have been
>> able to do this for a few of my repositories, but one in particular
>> isn't working. The dump file for this repository is > 9 GB. The load
>> command starts out, but never finishes. The size of my repository
>> gets to 2.1 GB and then it seems to stop even though the svnadmin load
>> command is still running. Eventually (> 24 hours) the command stops
>> (finishes) but the size of my repository is only 2.1 GB when it should
>> be around 9 GB. Also, not all of the files have been restored.
>>
>> Is there a limit in the size of a dump file that can be used? I would
>> be surprised if that limit were around 9 GB. I'm sure there are
>> repositories that are much larger than that.
>>
>> Any ideas on what could be wrong?
>
> Did you direct the svnadmin output to a file that you could tail and
> look for problems?
>
> FWIW, the final size of the repository ought to be significantly
> smaller than the dump file. I'd check for things like the final
> number of revisions. If the command did not throw an error, it
> probably worked.
I did redirect the output to a text file. The command never throws an
error, but never completes either. There should be 24 revisions in
the repository, but when I check out from the new server, there is
only 1, so something isn't working right.
Jeremy
Re: svn dump and load not preserving all files
Posted by Mark Phippard <ma...@gmail.com>.
On Tue, Jan 19, 2010 at 3:03 PM, Jeremy Conlin <jl...@gmail.com> wrote:
> I am migrating my repository to a new server. This requires that I
> dump the original and the create a new repository on the new server
> and use the svnadmin load command to import everything. I have been
> able to do this for a few of my repositories, but one in particular
> isn't working. The dump file for this repository is > 9 GB. The load
> command starts out, but never finishes. The size of my repository
> gets to 2.1 GB and then it seems to stop even though the svnadmin load
> command is still running. Eventually (> 24 hours) the command stops
> (finishes) but the size of my repository is only 2.1 GB when it should
> be around 9 GB. Also, not all of the files have been restored.
>
> Is there a limit in the size of a dump file that can be used? I would
> be surprised if that limit were around 9 GB. I'm sure there are
> repositories that are much larger than that.
>
> Any ideas on what could be wrong?
Did you direct the svnadmin output to a file that you could tail and
look for problems?
FWIW, the final size of the repository ought to be significantly
smaller than the dump file. I'd check for things like the final
number of revisions. If the command did not throw an error, it
probably worked.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
RE: svn dump and load not preserving all files
Posted by Thomas Loy <Th...@cbeyond.net>.
Which OS? Some operating systems have file size limits of 4 GB or less.
Cheers,
Tom Loy
-----Original Message-----
From: Jeremy Conlin [mailto:jlconlin@gmail.com]
Sent: Tuesday, January 19, 2010 3:03 PM
To: users@subversion.apache.org
Subject: svn dump and load not preserving all files
I am migrating my repository to a new server. This requires that I
dump the original and the create a new repository on the new server
and use the svnadmin load command to import everything. I have been
able to do this for a few of my repositories, but one in particular
isn't working. The dump file for this repository is > 9 GB. The load
command starts out, but never finishes. The size of my repository
gets to 2.1 GB and then it seems to stop even though the svnadmin load
command is still running. Eventually (> 24 hours) the command stops
(finishes) but the size of my repository is only 2.1 GB when it should
be around 9 GB. Also, not all of the files have been restored.
Is there a limit in the size of a dump file that can be used? I would
be surprised if that limit were around 9 GB. I'm sure there are
repositories that are much larger than that.
Any ideas on what could be wrong?
Thanks,
Jeremy