You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by "v.kalra.w" <vk...@agere.com> on 2005/09/27 20:18:49 UTC
svn import hangs in fs type repository
Hello:
We've been using subversion for some small projects and have been happy
enough with it that we want to move some of our big stuff under it.
Unfortunately, the very first such project has run into various
unexplained hangs at import time and some of the team members are having
second thoughts about it. I've googled for the scenarios and looked at
the mailing list archives, but I cannot seem to find anything similar
that others may have run across.
First the basics: We're trying to put about 10GB worth of data in about
6000 files under svn. The files are a mix of binary and ascii files,
mostly ascii. The repository type does NOT use Berkley DB -- We've tried
verisons 1.1.3 and 1.2.1, going over http or using the NFS file system
via the file:// mechanism, but in all cases, the import hangs part of
the way through the process. The whole process takes hours to go
through, so it is very frustrating to find it hang yet another time.
When using http, we're going to a machine running Solaris 8 running
Apache 2.0.54, but there's nothing in Apache logs to indicate any
problems.
Subversion 1.2.1 was compiled using BDB 4.2.52, though we're
using the FSFS backend.
The hangs do seem to happen moslty with somewhat large files (>10MB),
almost always binary (though not all the time). We were able to get it
all in once using *multiple* import statements - one per file (don't
ask), but realized afterwords that the various previous imports had
created .svn directories in the area being imported, and we ended up
importing all those in too, albeit with some error messages... :-(.
So we deleted that repository, tried again, only to get stuck with
importing stalling one more time.
We've requested our IT folks to install 1.2.3 to see if that would fix
the problems, but in the meantime, I'm hoping that I can get some help
here in trying to get this all to work. I only have a window of a few
more days before we throw up our collective hands and go back to
Synchronicity instead.
Any ideas?! Please help!
Thanks much
Vivek Kalra
vkalra@agere.com
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: svn import hangs in fs type repository
Posted by Ryan Schmidt <su...@ryandesign.com>.
On Sep 27, 2005, at 22:27, Mark Phippard wrote:
> "v.kalra.w" <vk...@agere.com> wrote on 09/27/2005 04:18:49 PM:
>
>> We've been using subversion for some small projects and have been
>> happy
>> enough with it that we want to move some of our big stuff under it.
>> Unfortunately, the very first such project has run into various
>> unexplained hangs at import time and some of the team members are
>> having
>> second thoughts about it. I've googled for the scenarios and
>> looked at
>> the mailing list archives, but I cannot seem to find anything similar
>> that others may have run across.
>>
>> First the basics: We're trying to put about 10GB worth of data in
>> about
>> 6000 files under svn. The files are a mix of binary and ascii files,
>> mostly ascii. The repository type does NOT use Berkley DB -- We've
>> tried
>> verisons 1.1.3 and 1.2.1, going over http or using the NFS file
>> system
>> via the file:// mechanism, but in all cases, the import hangs part of
>> the way through the process. The whole process takes hours to go
>> through, so it is very frustrating to find it hang yet another time.
>
> Depending on how your APR (on the server) was compiled it may not
> be able
> to handle files that are > 2GB. If you commit more than 2 GB at
> one time,
> you will have a revision file in the fsfs repository that APR cannot
> handle due to its size. I am not saying that is the problem, but
> that is
> somewhere to start.
I understood that APR 0.9 has this 2-GB limitation, and that APR 1.0
does not. And that if you want to move from APR 0.9 to 1.0, you
should dump your repositories first and reload after. Of course, for
you, since you don't have any repositories yet, you can forget that
step.
> So my recommendation would be to go back to the idea of breaking the
> import into chunks.
Since you're in a time crunch, maybe, I just hate workarounds. Seems
like it would be better in the long run to figure out why it's acting
this way and if there's a way to fix it. But if it's only the APR 0.9
2-GB revision-size limit, then this workaround (importing in smaller
chunks) should work and be safe.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: svn import hangs in fs type repository
Posted by Mark Phippard <Ma...@softlanding.com>.
"v.kalra.w" <vk...@agere.com> wrote on 09/28/2005 03:00:50 PM:
> On Tue, 27 Sep 2005, Mark Phippard wrote:
>
> %>>
> %>>Depending on how your APR (on the server) was compiled it may not be
able
> %>>to handle files that are > 2GB. If you commit more than 2 GB at one
time,
> %>>you will have a revision file in the fsfs repository that APR cannot
> %>>handle due to its size. I am not saying that is the problem, but
that is
> %>>somewhere to start.
> %>>
> Just so I understand this, does it mean that any one individual
> file from the N I'm importing cannot be more than 2GB, or does it
> mean that the total size of all N files that I'm importing cannot
> be more than 2GB.
If you have a version of APR with this problem (I think most/all 0.9.x
versions do) then it cannot handle an individual file that is > 2GB.
However, since fsfs stores each revision as a single file, you would get
this problem if a single commit resulted in a repository file > 2GB.
> OK, we're going to give this one more try -- we'll create a new
repository
> with just the tags, branches, and trunk directories in there, check out
> trunk, svn add directories, and check them in. Just to eliminate Apache
> out of the picture, we'll do this using the file:// URL and see if that
> works any better.
I think the main thing will be to avoid an individual commit that is >
2GB.
There is no problem with the total size of the repository being greater
than 2GB.
Mark
_____________________________________________________________________________
Scanned for SoftLanding Systems, Inc. and SoftLanding Europe Plc by IBM Email Security Management Services powered by MessageLabs.
_____________________________________________________________________________
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: svn import hangs in fs type repository
Posted by "v.kalra.w" <vk...@agere.com>.
On Tue, 27 Sep 2005, Mark Phippard wrote:
%>>
%>>Depending on how your APR (on the server) was compiled it may not be able
%>>to handle files that are > 2GB. If you commit more than 2 GB at one time,
%>>you will have a revision file in the fsfs repository that APR cannot
%>>handle due to its size. I am not saying that is the problem, but that is
%>>somewhere to start.
%>>
Just so I understand this, does it mean that any one individual
file from the N I'm importing cannot be more than 2GB, or does it
mean that the total size of all N files that I'm importing cannot
be more than 2GB.
%>>So my recommendation would be to go back to the idea of breaking the
%>>import into chunks.
%>>
I'm not sure if this is the problem -- for one, the import did seem to
go through *once*, though we ended up importing some of the subversion's
.svn directories in there as well. Also, I forgot to mention that we did
try to check in the files after just creating the repo instead of
importing them in, but with the same hang -- this was done over http,
not using file:// over NFS, and I don't know just what version of APR
was subversion compiled with.
%>>A lot of "seasoned" users try to avoid using svn import. An easy trick is
%>>to perform a checkout of your empty repository into a local folder. This
%>>then creates an empty working copy. You can then start creating your
%>>folder/file structure that you want and use svn add and svn commit to get
%>>it to the repository. This technique might make it easier for you to do
%>>this in chunks.
%>>
OK, we're going to give this one more try -- we'll create a new repository
with just the tags, branches, and trunk directories in there, check out
trunk, svn add directories, and check them in. Just to eliminate Apache
out of the picture, we'll do this using the file:// URL and see if that
works any better.
Thanks
Vivek
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: svn import hangs in fs type repository
Posted by Mark Phippard <Ma...@softlanding.com>.
"v.kalra.w" <vk...@agere.com> wrote on 09/27/2005 04:18:49 PM:
> We've been using subversion for some small projects and have been happy
> enough with it that we want to move some of our big stuff under it.
> Unfortunately, the very first such project has run into various
> unexplained hangs at import time and some of the team members are having
> second thoughts about it. I've googled for the scenarios and looked at
> the mailing list archives, but I cannot seem to find anything similar
> that others may have run across.
>
> First the basics: We're trying to put about 10GB worth of data in about
> 6000 files under svn. The files are a mix of binary and ascii files,
> mostly ascii. The repository type does NOT use Berkley DB -- We've tried
> verisons 1.1.3 and 1.2.1, going over http or using the NFS file system
> via the file:// mechanism, but in all cases, the import hangs part of
> the way through the process. The whole process takes hours to go
> through, so it is very frustrating to find it hang yet another time.
>
> Any ideas?! Please help!
Depending on how your APR (on the server) was compiled it may not be able
to handle files that are > 2GB. If you commit more than 2 GB at one time,
you will have a revision file in the fsfs repository that APR cannot
handle due to its size. I am not saying that is the problem, but that is
somewhere to start.
So my recommendation would be to go back to the idea of breaking the
import into chunks.
A lot of "seasoned" users try to avoid using svn import. An easy trick is
to perform a checkout of your empty repository into a local folder. This
then creates an empty working copy. You can then start creating your
folder/file structure that you want and use svn add and svn commit to get
it to the repository. This technique might make it easier for you to do
this in chunks.
Mark
_____________________________________________________________________________
Scanned for SoftLanding Systems, Inc. and SoftLanding Europe Plc by IBM Email Security Management Services powered by MessageLabs.
_____________________________________________________________________________
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org