You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Richard Lichvar <rl...@sainc.com> on 2009/07/14 14:36:22 UTC

Breaking-up Large Repository

First, I want to say how valuable the help has been from the people on
this list re. issues we've been having with SVN, particularly when the
requests are coming from an SVN newbie (actually a day-to-day general,
mainly Microsoft, sysadmin) who doesn't deal with SVN all the time.

 

Now, to the question at hand -

 

We've decided to break-up a very large repository (15+ GB) containing
multiple projects into separate repositories for related projects on a
new SVN server. We would like to have your advice as to the best way to
go about this. (The server is already up and running with some smaller
repositories that have already been successfully transitioned.) As
mentioned in other posts, one of the major challenges we have with the
old SVN server is a dire lack of disk space.

 

Many thanks in advance to those who will contribute their
experience/advice!

 

Rich Lichvar

------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2371239

To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].

RE: Breaking-up Large Repository

Posted by Giulio Troccoli <gi...@uk.linedata.com>.
>


Linedata Services (UK) Ltd
Registered Office: Bishopsgate Court, 4-12 Norton Folgate, London, E1 6DB
Registered in England and Wales No 3027851    VAT Reg No 778499447

-----Original Message-----


> From: Ryan Schmidt [mailto:subversion-2009b@ryandesign.com]
> Sent: 20 July 2009 23:20
> To: Giulio Troccoli
> Cc: Subversion Users
> Subject: Re: Breaking-up Large Repository
>
>
> On Jul 20, 2009, at 02:47, Giulio Troccoli wrote:
>
> >> On Jul 17, 2009, at 09:24, Giulio Troccoli wrote:
> >>
> >>>> On Jul 16, 2009, at 05:19, Giulio Troccoli wrote:
> >>>>
> >>>>> Thanks Ryan, but I actually meant what happens to the revision
> >>>>> number in case you use svnsync to synchronise just a
> part of the
> >>>>> repostiory, not when dumping, filtering and loading a
> repository.
> >>>>
> >>>>
> >>>> Oh. Of course. Um, I don't actually know. Could you do a
> test run
> >>>> and let us know? :)
> >>>
> >>> Well, I tried but it seems you can't do it.
> >>>
> >>> ln1sub01 svn> svnsync init http://ln1sub02/svn-util
> http://ln1sub01/
> >>> svn-util/trunk --username svnsync
> >>> svnsync: Session is rooted at
> >>> 'http://ln1sub01/svn-util/trunk' but the repos root is
> >>> 'http://ln1sub01/svn-util'
> >>
> >> Which version of svnsync is this?
> >>
> >> And which version of mod_dav_svn is being used to serve the
> >> repository?
> >
> > ln1sub01 svn> svnsync --version
> > svnsync, version 1.4.4 (r25188)
> >    compiled Jun 30 2007, 10:31:29
> >
> > ln1sub01 svn> rpm -qa | grep mod_dav_svn
> > mod_dav_svn-1.4.4-1
> >
> > Same for ln1sub02
>
> Sorry, I Cc'd the wrong mailing list before. Bringing this
> back to the Subversion Users list now.
>
> As you see, Subversion 1.4.4 is pretty old so I would
> recommend trying the latest version, currently 1.6.3, on both
> systems if possible.
>
> http://subversion.tigris.org/svn_1.6_releasenotes.html#svn-1.4-
> deprecation
>

I know, but we are having problems upgrading so I still used the old version. I will try again when the upgrade to 1.6.3 is done.

------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2372875

To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].


Re: Breaking-up Large Repository

Posted by Ryan Schmidt <su...@ryandesign.com>.
On Jul 20, 2009, at 02:47, Giulio Troccoli wrote:

>> On Jul 17, 2009, at 09:24, Giulio Troccoli wrote:
>>
>>>> On Jul 16, 2009, at 05:19, Giulio Troccoli wrote:
>>>>
>>>>> Thanks Ryan, but I actually meant what happens to the
>>>>> revision number in case you use svnsync to
>>>>> synchronise just a part of the repostiory, not when
>>>>> dumping, filtering and loading a repository.
>>>>
>>>>
>>>> Oh. Of course. Um, I don't actually know. Could you
>>>> do a test run and let us know? :)
>>>
>>> Well, I tried but it seems you can't do it.
>>>
>>> ln1sub01 svn> svnsync init http://ln1sub02/svn-util
>>> http://ln1sub01/
>>> svn-util/trunk --username svnsync
>>> svnsync: Session is rooted at
>>> 'http://ln1sub01/svn-util/trunk' but the
>>> repos root is 'http://ln1sub01/svn-util'
>>
>> Which version of svnsync is this?
>>
>> And which version of mod_dav_svn is being used to
>> serve the repository?
>
> ln1sub01 svn> svnsync --version
> svnsync, version 1.4.4 (r25188)
>    compiled Jun 30 2007, 10:31:29
>
> ln1sub01 svn> rpm -qa | grep mod_dav_svn
> mod_dav_svn-1.4.4-1
>
> Same for ln1sub02

Sorry, I Cc'd the wrong mailing list before. Bringing
this back to the Subversion Users list now.

As you see, Subversion 1.4.4 is pretty old so I
would recommend trying the latest version,
currently 1.6.3, on both systems if possible.

http://subversion.tigris.org/svn_1.6_releasenotes.html#svn-1.4- 
deprecation

------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2372749

To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].

Re: Breaking-up Large Repository

Posted by Ryan Schmidt <su...@ryandesign.com>.
On Jul 16, 2009, at 05:19, Giulio Troccoli wrote:

> Thanks Ryan, but I actually meant what happens to the revision  
> number in case you use svnsync to synchronise just a part of the  
> repostiory, not when dumping, filtering and loading a repository.

Oh. Of course. Um, I don't actually know. Could you do a test run and  
let us know? :)

------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2371958

To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].

RE: Breaking-up Large Repository

Posted by Giulio Troccoli <gi...@uk.linedata.com>.
>


Linedata Services (UK) Ltd
Registered Office: Bishopsgate Court, 4-12 Norton Folgate, London, E1 6DB
Registered in England and Wales No 3027851    VAT Reg No 778499447

-----Original Message-----


> From: Ryan Schmidt [mailto:subversion-2009b@ryandesign.com]
> Sent: 16 July 2009 01:09
> To: Giulio Troccoli
> Cc: users@subversion.tigris.org
> Subject: Re: Breaking-up Large Repository
>
> On Jul 15, 2009, at 03:58, Giulio Troccoli wrote:
>
> >> On Tue, Jul 14, 2009 at 10:35:26AM -0500, Ryan Schmidt wrote:
> >>
> >>> You can use svnsync to transfer all or part of the repository to
> >>> another server. So you can either sync the whole thing to another
> >>> server with more disk space and then do your dump, filter
> and load;
> >>> or you can sync just the project you want to extract to the new
> >>> server and there's your new repository all ready to go
> (well, you'd
> >>> have to copy over conf files and hook scripts if you have
> them). At
> >>> least, I was led to believe you can now, with the latest
> Subversion,
> >>> sync just a part of a repository; I'm not sure what the
> syntax for
> >>> that is.
> >>
> >> $ svnsync help init
> >> initialize (init): usage: svnsync initialize DEST_URL SOURCE_URL
> >>
> >> [...]
> >>
> >> If the source URL is not the root of a repository, only
> the specified
> >> part of the repository will be synchronized.
> >
> > What would happen to the revision numbers? Will I have
> empty revisions
> > in the destination repository? Or will the revision be
> sequential and
> > therefore not in sync with the source repository?
>
>
> svndumpfilter has the options --drop-empty-revs and
> --renumber-revs to help you choose which of those behaviors you want.
>
> Um, I believe if you use neither option, the default is to
> keep your revision numbers as they were, and thus to insert
> empty revisions for the ones that are not relevant to your
> new thinned repository. If you want the revisions renumbered
> / compacted, then you use both -- renumber-revs and
> --drop-empty-revs. I'm not entirely sure what happens if you
> just use the one switch or the other, so I'm not sure why
> there are two switches.

Thanks Ryan, but I actually meant what happens to the revision number in case you use svnsync to synchronise just a part of the repostiory, not when dumping, filtering and loading a repository.

------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2371710

To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].


Re: Breaking-up Large Repository

Posted by Ryan Schmidt <su...@ryandesign.com>.
On Jul 15, 2009, at 03:58, Giulio Troccoli wrote:

>> On Tue, Jul 14, 2009 at 10:35:26AM -0500, Ryan Schmidt wrote:
>>
>>> You can use svnsync to transfer all or part of the repository to
>>> another server. So you can either sync the whole thing to another
>>> server with more disk space and then do your dump, filter and  
>>> load; or
>>> you can sync just the project you want to extract to the new server
>>> and there's your new repository all ready to go (well, you'd have to
>>> copy over conf files and hook scripts if you have them). At least, I
>>> was led to believe you can now, with the latest Subversion, sync  
>>> just
>>> a part of a repository; I'm not sure what the syntax for that is.
>>
>> $ svnsync help init
>> initialize (init): usage: svnsync initialize DEST_URL SOURCE_URL
>>
>> [...]
>>
>> If the source URL is not the root of a repository, only the
>> specified part of the repository will be synchronized.
>
> What would happen to the revision numbers? Will I have empty  
> revisions in the destination repository? Or will the revision be  
> sequential and therefore not in sync with the source repository?


svndumpfilter has the options --drop-empty-revs and --renumber-revs  
to help you choose which of those behaviors you want.

Um, I believe if you use neither option, the default is to keep your  
revision numbers as they were, and thus to insert empty revisions for  
the ones that are not relevant to your new thinned repository. If you  
want the revisions renumbered / compacted, then you use both -- 
renumber-revs and --drop-empty-revs. I'm not entirely sure what  
happens if you just use the one switch or the other, so I'm not sure  
why there are two switches.

------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2371644

To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].

RE: Breaking-up Large Repository

Posted by Giulio Troccoli <gi...@uk.linedata.com>.
>


Linedata Services (UK) Ltd
Registered Office: Bishopsgate Court, 4-12 Norton Folgate, London, E1 6DB
Registered in England and Wales No 3027851    VAT Reg No 778499447

-----Original Message-----


> From: Stefan Sperling [mailto:stsp@elego.de]
> Sent: 14 July 2009 16:48
> To: Ryan Schmidt
> Cc: Richard Lichvar; users@subversion.tigris.org
> Subject: Re: Breaking-up Large Repository
>
> On Tue, Jul 14, 2009 at 10:35:26AM -0500, Ryan Schmidt wrote:
> > You can use svnsync to transfer all or part of the repository to
> > another server. So you can either sync the whole thing to another
> > server with more disk space and then do your dump, filter
> and load; or
> > you can sync just the project you want to extract to the new server
> > and there's your new repository all ready to go (well,
> you'd have to
> > copy over conf files and hook scripts if you have them). At
> least, I
> > was led to believe you can now, with the latest Subversion,
> sync just
> > a part of a repository; I'm not sure what the syntax for that is.
>
>  $ svnsync help init
>  initialize (init): usage: svnsync initialize DEST_URL SOURCE_URL
>
>  [...]
>
>  If the source URL is not the root of a repository, only the
> specified part of the repository will be synchronized.
>

What would happen to the revision numbers? Will I have empty revisions in the destination repository? Or will the revision be sequential and therefore not in sync with the source repository?

------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2371423

To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].


Re: Breaking-up Large Repository

Posted by Les Mikesell <le...@gmail.com>.
Ryan Schmidt wrote:
> 
>> Fantastic! Now, if I wanted to migrate 2 projects from a repository  
>> that
>> held 3, would it be acceptable to call svnsync twice using the same
>> destination repository (obviously with a corresponding folder  
>> structure
>> in the dest_url)?
> 
> You mean pull two projects from the old repository into a single new  
> repository? I don't know but I would not expect that to be possible  
> with svnsync. For that case, you would need to dump, filter (to  
> include both projects), and load.

Yes, dump/filter/load will let you include or exclude parts to separate 
things the way you want.  Even with that you may have some problems with 
items that were originally created in a location you don't want to 
include but were later copied into the space you want.

-- 
   Les Mikesell
    lesmikesell@gmail.com

------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2371278

To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].

Re: Breaking-up Large Repository

Posted by Ryan Schmidt <su...@ryandesign.com>.
On Jul 14, 2009, at 10:56, Bolstridge, Andrew wrote:

>> On Tue, Jul 14, 2009 at 10:35:26AM -0500, Ryan Schmidt wrote:
>>> ...or you can sync just the project you want to extract to the new
>>> server and there's your new repository all ready to go...
>>
>>  $ svnsync help init
>>  initialize (init): usage: svnsync initialize DEST_URL SOURCE_URL
>>
>>  [...]
>>
>>  If the source URL is not the root of a repository, only the
>>  specified part of the repository will be synchronized.
>>
> Fantastic! Now, if I wanted to migrate 2 projects from a repository  
> that
> held 3, would it be acceptable to call svnsync twice using the same
> destination repository (obviously with a corresponding folder  
> structure
> in the dest_url)?

You mean pull two projects from the old repository into a single new  
repository? I don't know but I would not expect that to be possible  
with svnsync. For that case, you would need to dump, filter (to  
include both projects), and load.

------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2371272

To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].

RE: Breaking-up Large Repository

Posted by "Bolstridge, Andrew" <an...@intergraph.com>.
> -----Original Message-----
> From: Stefan Sperling [mailto:stsp@elego.de]
> Sent: Tuesday, July 14, 2009 4:48 PM
> To: Ryan Schmidt
> Cc: Richard Lichvar; users@subversion.tigris.org
> Subject: Re: Breaking-up Large Repository
> 
> On Tue, Jul 14, 2009 at 10:35:26AM -0500, Ryan Schmidt wrote:
> > ...or you can sync just the project you want to extract to the new
> > server and there's your new repository all ready to go...
> 
>  $ svnsync help init
>  initialize (init): usage: svnsync initialize DEST_URL SOURCE_URL
> 
>  [...]
> 
>  If the source URL is not the root of a repository, only the
>  specified part of the repository will be synchronized.
> 
Fantastic! Now, if I wanted to migrate 2 projects from a repository that
held 3, would it be acceptable to call svnsync twice using the same
destination repository (obviously with a corresponding folder structure
in the dest_url)?

------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2371266

To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].


Re: Breaking-up Large Repository

Posted by Stefan Sperling <st...@elego.de>.
On Tue, Jul 14, 2009 at 10:35:26AM -0500, Ryan Schmidt wrote:
> You can use svnsync to transfer all or part of the repository to  
> another server. So you can either sync the whole thing to another  
> server with more disk space and then do your dump, filter and load;  
> or you can sync just the project you want to extract to the new  
> server and there's your new repository all ready to go (well, you'd  
> have to copy over conf files and hook scripts if you have them). At  
> least, I was led to believe you can now, with the latest Subversion,  
> sync just a part of a repository; I'm not sure what the syntax for  
> that is.

 $ svnsync help init
 initialize (init): usage: svnsync initialize DEST_URL SOURCE_URL
 
 [...]
 
 If the source URL is not the root of a repository, only the
 specified part of the repository will be synchronized.

 [...]

Should work with 1.6.x at least.

Note: If you can, use 1.6.3. It contains a fix for a rather annoying
bug in svnsync affecting everyone who does not have their log messages
etc. in clean UTF-8.

Stefan

Re: Breaking-up Large Repository

Posted by Ryan Schmidt <su...@ryandesign.com>.
On Jul 14, 2009, at 09:36, Richard Lichvar wrote:

> First, I want to say how valuable the help has been from the people  
> on this list re. issues we’ve been having with SVN, particularly  
> when the requests are coming from an SVN newbie (actually a day-to- 
> day general, mainly Microsoft, sysadmin) who doesn’t deal with SVN  
> all the time.
>
> Now, to the question at hand –
>
> We’ve decided to break-up a very large repository (15+ GB)  
> containing multiple projects into separate repositories for related  
> projects on a new SVN server. We would like to have your advice as  
> to the best way to go about this. (The server is already up and  
> running with some smaller repositories that have already been  
> successfully transitioned.) As mentioned in other posts, one of the  
> major challenges we have with the old SVN server is a dire lack of  
> disk space.

You can use svnsync to transfer all or part of the repository to  
another server. So you can either sync the whole thing to another  
server with more disk space and then do your dump, filter and load;  
or you can sync just the project you want to extract to the new  
server and there's your new repository all ready to go (well, you'd  
have to copy over conf files and hook scripts if you have them). At  
least, I was led to believe you can now, with the latest Subversion,  
sync just a part of a repository; I'm not sure what the syntax for  
that is.

------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2371254

To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].