Re: Breaking up a monolothic repository

[Ullrich Jans <ul...@elektrobit.com>]

Am 10.09.2013 19:45, schrieb Thomas Harold:

> When we moved from a monolithic repository to per-client repositories a
> few years ago, we went ahead and:
>
> - Rebased the paths up one or two levels (old system was something like
> "monolithicrepo/[a-z]/[client directories]/[job directory]") so that the
> urls were now "clientrepo/[job directory]".  That was a tricky thing to
> do and we had to 'sed' the output of the dump filter before importing it
> back.
>
> It broke a few things, such as svn:externals which were not
> relative-pathed, but was worth it in the long run so that our URLs got
> shorter.
>
> - Made sure that the new repos all had unique UUIDs.
>
> - Renumbered all of the resulting revisions as we loaded things back in.
>   But we didn't have to deal with any bug tracking systems that referred
> to a specific revision.  And having lower revision numbers was
> preferred, along with dropping revisions that referred to other projects.

I'm now facing the same problem. My users want the rebasing, but during 
the dump/load instead of after the fact (apparently, it causes issues 
with their environment when they need to go back to an earlier revision 
to reproduce something). They also want to keep the empty revisions (for 
references from the issue tracker).

I haven't tried it with svnadmin dump followed by svndumpfilter (I don't 
think it has that capability).

I've tried svnrdump (from svn 1.7), it resulted in either a new 
repository with the full path included (rdump/load all revs) or an 
interesting failure mode with a missing node during a copy operation 
when rdump -r <revision_after_path>:HEAD was used

I've also tried using svnsync, but that also results in the full path 
included, no rebasing.

How did you do it? Also, am I missing something that has been included 
in a current svn version?

Cheers,

Ulli

-- 
Ullrich Jans, Specialist, IT-A
Phone: +49 9131 7701-6627, mailto:ullrich.jans@elektrobit.com
Fax: +49 9131 7701-6333, www.elektrobit.com

Elektrobit Automotive GmbH, Am Wolfsmantel 46, 91058 Erlangen, Germany
Managing Directors: Alexander Kocher, Gregor Zink
Register Court Fürth HRB 4886


----------------------------------------------------------------
Please note: This e-mail may contain confidential information
intended solely for the addressee. If you have received this
e-mail in error, please do not disclose it to anyone, notify
the sender promptly, and delete the message from your system.
Thank you.

RE: Breaking up a monolothic repository

[Bob Archer <Bo...@amsi.com>]

> Am 10.09.2013 19:45, schrieb Thomas Harold:
> 
> > When we moved from a monolithic repository to per-client repositories
> > a few years ago, we went ahead and:
> >
> > - Rebased the paths up one or two levels (old system was something
> > like "monolithicrepo/[a-z]/[client directories]/[job directory]") so
> > that the urls were now "clientrepo/[job directory]".  That was a
> > tricky thing to do and we had to 'sed' the output of the dump filter
> > before importing it back.
> >
> > It broke a few things, such as svn:externals which were not
> > relative-pathed, but was worth it in the long run so that our URLs got
> > shorter.
> >
> > - Made sure that the new repos all had unique UUIDs.
> >
> > - Renumbered all of the resulting revisions as we loaded things back in.
> >   But we didn't have to deal with any bug tracking systems that
> > referred to a specific revision.  And having lower revision numbers
> > was preferred, along with dropping revisions that referred to other projects.
> 
> I'm now facing the same problem. My users want the rebasing, but during the
> dump/load instead of after the fact (apparently, it causes issues with their
> environment when they need to go back to an earlier revision to reproduce
> something). They also want to keep the empty revisions (for references from
> the issue tracker).

Wouldn't it be much simpler to keep the current repository as a read only archives and move the HEAD of each project into its own repo?


> I haven't tried it with svnadmin dump followed by svndumpfilter (I don't think it
> has that capability).
> 
> I've tried svnrdump (from svn 1.7), it resulted in either a new repository with
> the full path included (rdump/load all revs) or an interesting failure mode with
> a missing node during a copy operation when rdump -r
> <revision_after_path>:HEAD was used
> 
> I've also tried using svnsync, but that also results in the full path included, no
> rebasing.
> 
> How did you do it? Also, am I missing something that has been included in a
> current svn version?
> 
> Cheers,
> 
> Ulli

Re: Breaking up a monolothic repository

[Thomas Harold <th...@nybeta.com>]

On 10/2/2013 10:36 AM, Ullrich Jans wrote:
>
> I'm now facing the same problem. My users want the rebasing, but during
> the dump/load instead of after the fact (apparently, it causes issues
> with their environment when they need to go back to an earlier revision
> to reproduce something). They also want to keep the empty revisions (for
> references from the issue tracker).
>
> I haven't tried it with svnadmin dump followed by svndumpfilter (I don't
> think it has that capability).

The command we ended up using back in May 2011 when we did this looked 
like the following.  It's been two years, but I'm pretty sure these two 
scripts is all we ended up using.

- We had a master dump of the entire brc-jobs repository.
- Target repository name was "brc-jobs-zp" (CLCODE)
- It takes the dump and splits it into a smaller chunk (CLPATH).
- Had to edit the script for each new client/path that we wanted to 
split out.

It does *not* attempt to rebase the individual projects up to the root 
directory.  It *is* possible by using 'sed' to do this in the resulting 
dump file, but it is trick.

----------------------------------------
#!/bin/bash

SOURCE=/mnt/scratch/svn-dump-brc-jobs.may2011.dump.gz

DESTDIR=/var/svn/
DESTPFX=svn-raw-brc-jobs-
DESTSFX=10xx.dump.gz

CLCODE=zp
CLPATH=Z/ZP_SingleJobs

SDFOPTS='--drop-empty-revs  --renumber-revs'

date

echo ${DESTDIR}${DESTPFX}${CLCODE}${DESTSFX}

svnadmin dump --quiet /var/svn/brc-jobs | \
     svndumpfilter include --quiet $SDFOPTS $CLPATH | \
     gzip > ${DESTDIR}${DESTPFX}${CLCODE}${DESTSFX}

date
----------------------------------------

The mirror to this was the script that created the new SVN repository 
and loads in the individual dump.

Note the commented out 'sed' lines where we attempted to rebase 
individual project folders back up to the root of the repository.  They 
didn't work, so we ended up just doing a move operation in the 
TortoiseSVN repository browser.

- It changes the UUID of the newly created repository to be something 
unique instead of using the old repo's UUID.
- Had to be edited anew for each new client/path.

----------------------------------------
#!/bin/bash

SRCDIR=/var/svn/
SRCPFX=svn-raw-brc-jobs-
SRCSFX=10xx.dump.gz

DESTDIR=/var/svn/
DESTPFX=svn-newbase-brc-jobs-
DESTSFX=10xx.dump.gz

SDFOPTS='--quiet --drop-empty-revs  --renumber-revs'

CLPARENT=Z
CLCODE=zp

date

#gunzip -c ${SRCDIR}${SRCPFX}${CLCODE}${SRCSFX} | \
#sed "s/Node-path: $CLPATH\//Node-path: /" | \
#sed "s/Node-copyfrompath: $CLPATH\//Node-copyfrompath: /" | \
#gzip > ${DESTDIR}${DESTPFX}${CLCODE}${DESTSFX}

svn mkdir -m "Import from brc-jobs" 
file:///var/svn/brc-jobs-${CLCODE}/${CLPARENT}

gunzip -c ${SRCDIR}${SRCPFX}${CLCODE}${SRCSFX} | \
   svnadmin load --quiet /var/svn/brc-jobs-${CLCODE}

svnlook uuid /var/svn/brc-jobs-${CLCODE}
svnadmin setuuid /var/svn/brc-jobs-${CLCODE}
svnlook uuid /var/svn/brc-jobs-${CLCODE}
svnadmin pack /var/svn/brc-jobs-${CLCODE}

chmod -R 775 /var/svn/brc-jobs-${CLCODE}
chmod -R g+s /var/svn/brc-jobs-${CLCODE}/db
chgrp -R svn-brc-jobs /var/svn/brc-jobs-${CLCODE}

date
----------------------------------------

I do wish I could have figured out the 'sed' commands to move a project 
from /Z/ZP_SingleJobs/JOBNR to be just /JOBNR in the repository, but 
there wasn't time.

For rebasing, that's probably your missing piece... which I don't have.