Copying from one repository to another, flattening the early part of the source repo's history in the target repo?

[Dan Stromberg <ds...@gmail.com>]

I'm working on some Subversion-related software for propagating trunk
checkins to any of dozens of target branches, based on trunk's checkin
comments (that is, where the checkins need to subsequently go automatically is
determined by the checkin comments in trunk, and those comments are
prepared pretty consistently by a custom GUI; the GUI piece is already
written and in production).

I of course want it to work well when we deploy it - and the trunk
we'll ultimately be running it against is rather huge.

So to make it work well from the start, I need to test it well.  To
test it, I really want to copy from one SVN server's trunk to another
SVN server's repository - flattening a significant part of the early
history from the source trunk, and then having as-identical-as-practical
checkin comments for the later history.

Is there already a way of setting up such a test branch?  I've googled
about it quite a bit and come up with things that are close (svnsync:
way too much test data, svn merge: too few checkin comments unless I
do them individually somehow), but not quite what I need.  It's
seeming like writing the program to prepare the test environment is
nearly as complex as what I'm intending to test.

Please note that although I've been an SVN admin in the past, I do not
have administrative access to these SVN servers.

TIA!

(PS: I've written a bunch of unit tests for this, but that's just not
enough - it needs to be tested as a complete system too with
minimal deviation from how it's to be used In Real Life)

Re: Copying from one repository to another, flattening the early part of the source repo's history in the target repo?

[Dan Stromberg <ds...@gmail.com>]

On Thu, Apr 29, 2010 at 11:10 AM, Tyler Roscoe <ty...@cryptio.net> wrote:
> On Thu, Apr 29, 2010 at 11:06:40AM -0700, Dan Stromberg wrote:
>> So to make it work well from the start, I need to test it well.  To
>> test it, I really want to copy from one SVN server's trunk to another
>> SVN server's repository - flattening a significant part of the early
>> history from the source trunk, and then having as-identical-as-practical
>> checkin comments for the later history.
>
> Use svnadmin dump and maybe svnfilter to generate a dump file from your
> production repo.
>
> Use svnadmin load on the dump file to load this massaged repo data into
> your test repo.
>
> Bob's your uncle.
>
> tyler
>

I suppose if I had svn server access, I could svn export an oldish
revision (n), svn import to the new server, and then use svnadmin dump
with -r n+1:HEAD and svnadmin load on the new server...  But I don't
have svn server access in this case - to either server.  I could set
one up, but then I'd only have access to part of the picture, and I'm
not sure I have enough disk space if I try.

Is there any other way?

I might be able to coax an svn admin at this site to help, but that's
relatively likely to hit some red tape.  Is there a way of doing this
without server access?

Re: Copying from one repository to another, flattening the early part of the source repo's history in the target repo?

[Tyler Roscoe <ty...@cryptio.net>]

On Thu, Apr 29, 2010 at 11:06:40AM -0700, Dan Stromberg wrote:
> So to make it work well from the start, I need to test it well.  To
> test it, I really want to copy from one SVN server's trunk to another
> SVN server's repository - flattening a significant part of the early
> history from the source trunk, and then having as-identical-as-practical
> checkin comments for the later history.

Use svnadmin dump and maybe svnfilter to generate a dump file from your
production repo.

Use svnadmin load on the dump file to load this massaged repo data into
your test repo.

Bob's your uncle.

tyler

Re: Copying from one repository to another, flattening the early part of the source repo's history in the target repo?

[Dan Stromberg <ds...@gmail.com>]

On Thu, Apr 29, 2010 at 11:23 AM, Les Mikesell <le...@gmail.com> wrote:
> On 4/29/2010 1:06 PM, Dan Stromberg wrote:
>>
>> I'm working on some Subversion-related software for propagating trunk
>> checkins to any of dozens of target branches, based on trunk's checkin
>> comments (that is, where the checkins need to subsequently go
>> automatically is
>> determined by the checkin comments in trunk, and those comments are
>> prepared pretty consistently by a custom GUI; the GUI piece is already
>> written and in production).
>>
>> I of course want it to work well when we deploy it - and the trunk
>> we'll ultimately be running it against is rather huge.
>>
>> So to make it work well from the start, I need to test it well.  To
>> test it, I really want to copy from one SVN server's trunk to another
>> SVN server's repository - flattening a significant part of the early
>> history from the source trunk, and then having as-identical-as-practical
>> checkin comments for the later history.
>
> I don't understand the need to flatten the trunk.  Fanning out from the
> trunk to many branches should be straightforward enough, fhough.

The source branch is terabytes of source code.  I want to restrict by
subdirectory, and flatten history to reduce the storage requirements.

As far as straightforward enough, yes, it's conceptually simple enough,
though I've got 600+ lines of Python code written for this so far (not
including unit tests or test inputs), and it's not done yet.  It includes XML
parsing, YAML parsing, SVN interaction (with error checking), a flat file
database module (because the customer doesn't want to use gdbm or
similar), etc.

>> Is there already a way of setting up such a test branch?  I've googled
>> about it quite a bit and come up with things that are close (svnsync:
>> way too much test data, svn merge: too few checkin comments unless I
>> do them individually somehow), but not quite what I need.  It's
>> seeming like writing the program to prepare the test environment is
>> nearly as complex as what I'm intending to test.
>>
>> Please note that although I've been an SVN admin in the past, I do not
>> have administrative access to these SVN servers.
>
> The only way you can modify existing history is with a svnadmin
> dump/filter/load and you need admin access for that.   Can you svnsync the
> live repo to one you control, then dump from there to get a base for your
> test?

I guess I'm not entirely looking to modify existing history, so much as compress
the old stuff by flattening and restricting it, and then cloning the
recent history.
I can imagine a program to do this (at least to an extent) that doesn't require
admin access, but I'm hoping someone's already written it.

It's starting to sound like I'm going to need to write a bash script
to handle this.

Re: Copying from one repository to another, flattening the early part of the source repo's history in the target repo?

[Les Mikesell <le...@gmail.com>]

On 4/29/2010 1:06 PM, Dan Stromberg wrote:
> I'm working on some Subversion-related software for propagating trunk
> checkins to any of dozens of target branches, based on trunk's checkin
> comments (that is, where the checkins need to subsequently go automatically is
> determined by the checkin comments in trunk, and those comments are
> prepared pretty consistently by a custom GUI; the GUI piece is already
> written and in production).
>
> I of course want it to work well when we deploy it - and the trunk
> we'll ultimately be running it against is rather huge.
>
> So to make it work well from the start, I need to test it well.  To
> test it, I really want to copy from one SVN server's trunk to another
> SVN server's repository - flattening a significant part of the early
> history from the source trunk, and then having as-identical-as-practical
> checkin comments for the later history.

I don't understand the need to flatten the trunk.  Fanning out from the 
trunk to many branches should be straightforward enough, fhough.

> Is there already a way of setting up such a test branch?  I've googled
> about it quite a bit and come up with things that are close (svnsync:
> way too much test data, svn merge: too few checkin comments unless I
> do them individually somehow), but not quite what I need.  It's
> seeming like writing the program to prepare the test environment is
> nearly as complex as what I'm intending to test.
>
> Please note that although I've been an SVN admin in the past, I do not
> have administrative access to these SVN servers.

The only way you can modify existing history is with a svnadmin 
dump/filter/load and you need admin access for that.   Can you svnsync 
the live repo to one you control, then dump from there to get a base for 
your test?

-- 
   Les Mikesell
    lesmikesell@gmail.com