Proposal for a new exporter tool

[Ramkumar Ramachandra <ar...@gmail.com>]

Hi,

I have recently developed a new svn_editor_t editor for Subversion
that uses the replay API to generate a dumpfile v3 over the network
on-the-fly (without touching the filesystem except for some temporary
files). I initially started building the tool to merge it into the
git.git tree and link it to a "remote helper" tool to allow Git to
seamlessly import Subversion history as part of a Google Summer of
Code project. I later figured that the tool would be useful to both
SVN server admins to generate backups and other VCS developers; my
current plan is to get it merged it into git.git, and then (if it's a
good idea) get it merged into the Subversion tree and remove it from
git.git. As an added bonus, it currently performs significantly better
than `svnsync` because it doesn't need to touch the filesystem or
apply any deltas.

So far, I've only been developing it as an independent project, and I
posted a series reporting my progress last night [1]. The series
includes a nice validation script. I'd really appreciate it if you
could review, test, and deem the series fit for the Subversion trunk.

So, is it a good idea to merge it into the Subversion tree? If it is,
I can re-license the project, generate patches against the Subversion
tree excluding debug_editor.c, LICENSE, validation.sh and other noise
to get the merge process started. I can probably even get it to plug
into `svnsync` as another editor.

Thanks.

-- Ram

[1]: http://thread.gmane.org/gmane.comp.version-control.subversion.devel/120915

Re: Proposal for a new exporter tool

[Julian Foad <ju...@wandisco.com>]

Stefan Sperling wrote:
> Ramkumar Ramachandra wrote:
> > I have recently developed a new svn_editor_t editor for Subversion
> > that uses the replay API to generate a dumpfile v3 over the network
[...]

> I see value in adding this feature to Subversion.
> Generating a dump file from a remote repository certainly has a
> couple of interesting use cases.

It's very nice to see someone coming and enthusiastically offering us
some new working code.

I agree it looks like it's useful as well as interesting, and it seems
to me that this would be a reasonable extension to the set of software
that is "the Subversion software" - it implements a sensible combination
of existing concepts.

> I'd prefer adding this as a new command, such as "svnrdump" ("remote dump").
> 
> Obviously, we'll need a patch in proper shape. That means it's against
> our trunk (it doesn't matter whether you produce diff output with git or
> with svn), and that the coding style (indentation etc.) matches the
> conventions used in svn, and that the tool behaves like the other CLI
> tools we have regarding option parsing, help output, etc.
> 
> Also, please don't send one patch per file, but send the entire thing
> in one patch.

I'll just point out that his patches yesterday were each one logical
change made across several files, in case you missed that.

- Julian


> A set of unit tests for the new command within our regression test
> suite would be nice (similar to svnlook_tests.py and svnsync_tests.py).
> These could be submitted in a separate patch.
> 
> Unless you hear objections during the next few days, I'd suggest you
> start working on a patch aimed at our trunk (instead of at git).
> 
> I'm not sure how GSOC handles cases like this though. Please make sure
> your mentor is aware of patches you send here, and is willing to work
> with you even if you direct them at Subversion rather than git.
> I don't expect a problem (being a gsoc mentor myself this year and last
> year I have had good experience with the program and the people who
> run it), but you should certainly make sure Google is still willing to
> support you even if you technically switch projects while summer of code
> is running.
> 
> Thanks,
> Stefan

Re: Proposal for a new exporter tool

[Stefan Sperling <st...@elego.de>]

On Wed, Jul 07, 2010 at 06:31:48PM +0200, Ramkumar Ramachandra wrote:
> > On Wed, Jul 07, 2010 at 04:47:33PM +0200, Ramkumar Ramachandra wrote:
> > Also, please don't send one patch per file, but send the entire thing
> > in one patch.
> 
> Okay. It'll probably be a small patch with just dump_editor.c.

I'm picturing something like:

  subversion/svnrdump/
  subversion/svnrdump/svnrdump.c  <-- main() program
  subversion/svnrdump/dump-editor.c  <-- the editor

plus any other support files you need in that directory.

> > A set of unit tests for the new command within our regression test
> > suite would be nice (similar to svnlook_tests.py and svnsync_tests.py).
> > These could be submitted in a separate patch.
> 
> Okay. Will do after the editor patch.

Great.

> > Unless you hear objections during the next few days, I'd suggest you
> > start working on a patch aimed at our trunk (instead of at git).
> 
> Merging it into git.git is almost effortless from here, and I'll have
> the added benefit of more people looking at the code and testing
> it. I'll remove it from git.git when it's merged into Subversion.

Fair enough.

Stefan

Re: Proposal for a new exporter tool

[Ramkumar Ramachandra <ar...@gmail.com>]

Hi Daniel,

Daniel Shahaf writes:
> Ramkumar Ramachandra wrote on Wed, 7 Jul 2010 at 18:31 +0200:
> > Stefan Sperling writes:
> > > I'd prefer adding this as a new command, such as "svnrdump" ("remote dump").
> > 
> > Actually, since I can never support dumpfilev2 with this tool, it
> > probably doesn't make sense to plug it into `svnsync`.
> 
> I don't see the connection?

Well, the --deltas flag is implied for starters. Also a lot of the
delta application code is pretty unrelated. I suppose this isn't a big
deal anyway.

> (am I against putting this in svnsync?  I didn't think I was when
> I started this mail, but having finished I think I am)

Okay, I'll implement it as a separate command as Stefan suggested.

-- Ram

Re: Proposal for a new exporter tool

[Daniel Shahaf <d....@daniel.shahaf.name>]

Ramkumar Ramachandra wrote on Wed, 7 Jul 2010 at 18:31 +0200:
> Stefan Sperling writes:
> > I'd prefer adding this as a new command, such as "svnrdump" ("remote dump").
> 
> Actually, since I can never support dumpfilev2 with this tool, it
> probably doesn't make sense to plug it into `svnsync`.

I don't see the connection?

Personally, I'm worried about the more incremental aspects of svnsync:
e.g., when only half of the revisions are created via svnsync (and the
other half via emailing dump files), or when 'copy-revprops' is used,
or the ability to just swap source and target repositories (after fixing
a couple of revprops), etc.  If you plug into svnsync, you have to
implement all these, or special-case them...

Daniel
(am I against putting this in svnsync?  I didn't think I was when
I started this mail, but having finished I think I am)

Re: Proposal for a new exporter tool

[Ramkumar Ramachandra <ar...@gmail.com>]

Hi Stefan,

Stefan Sperling writes:
> On Wed, Jul 07, 2010 at 04:47:33PM +0200, Ramkumar Ramachandra wrote:
> > Hi,
> > 
> > I have recently developed a new svn_editor_t editor for Subversion
> > that uses the replay API to generate a dumpfile v3 over the network
> > on-the-fly (without touching the filesystem except for some temporary
> > files). I initially started building the tool to merge it into the
> > git.git tree and link it to a "remote helper" tool to allow Git to
> > seamlessly import Subversion history as part of a Google Summer of
> > Code project. I later figured that the tool would be useful to both
> > SVN server admins to generate backups and other VCS developers; my
> > current plan is to get it merged it into git.git, and then (if it's a
> > good idea) get it merged into the Subversion tree and remove it from
> > git.git. As an added bonus, it currently performs significantly better
> > than `svnsync` because it doesn't need to touch the filesystem or
> > apply any deltas.
> > 
> > So far, I've only been developing it as an independent project, and I
> > posted a series reporting my progress last night [1]. The series
> > includes a nice validation script. I'd really appreciate it if you
> > could review, test, and deem the series fit for the Subversion trunk.
> > 
> > So, is it a good idea to merge it into the Subversion tree? If it is,
> > I can re-license the project, generate patches against the Subversion
> > tree excluding debug_editor.c, LICENSE, validation.sh and other noise
> > to get the merge process started. I can probably even get it to plug
> > into `svnsync` as another editor.
> 
> I see value in adding this feature to Subversion.
> Generating a dump file from a remote repository certainly has a
> couple of interesting use cases.

Right, thanks.

> I'd prefer adding this as a new command, such as "svnrdump" ("remote dump").

Actually, since I can never support dumpfilev2 with this tool, it
probably doesn't make sense to plug it into `svnsync`.

> Obviously, we'll need a patch in proper shape. That means it's against
> our trunk (it doesn't matter whether you produce diff output with git or
> with svn), and that the coding style (indentation etc.) matches the
> conventions used in svn, and that the tool behaves like the other CLI
> tools we have regarding option parsing, help output, etc.

Ofcourse. I'll just ask my editor to convert it.

> Also, please don't send one patch per file, but send the entire thing
> in one patch.

Okay. It'll probably be a small patch with just dump_editor.c.

> A set of unit tests for the new command within our regression test
> suite would be nice (similar to svnlook_tests.py and svnsync_tests.py).
> These could be submitted in a separate patch.

Okay. Will do after the editor patch.

> Unless you hear objections during the next few days, I'd suggest you
> start working on a patch aimed at our trunk (instead of at git).

Merging it into git.git is almost effortless from here, and I'll have
the added benefit of more people looking at the code and testing
it. I'll remove it from git.git when it's merged into Subversion. I
estimate that I should have a patch ready to submit in a few days.

> I'm not sure how GSOC handles cases like this though. Please make sure
> your mentor is aware of patches you send here, and is willing to work
> with you even if you direct them at Subversion rather than git.
> I don't expect a problem (being a gsoc mentor myself this year and last
> year I have had good experience with the program and the people who
> run it), but you should certainly make sure Google is still willing to
> support you even if you technically switch projects while summer of code
> is running.

Don't worry about this. As far as Git is concerned, this is
technically NOT part of the GSoC- I've talked to the Git people about
this already.

-- Ram

Re: Proposal for a new exporter tool

[Stefan Sperling <st...@elego.de>]

On Wed, Jul 07, 2010 at 04:47:33PM +0200, Ramkumar Ramachandra wrote:
> Hi,
> 
> I have recently developed a new svn_editor_t editor for Subversion
> that uses the replay API to generate a dumpfile v3 over the network
> on-the-fly (without touching the filesystem except for some temporary
> files). I initially started building the tool to merge it into the
> git.git tree and link it to a "remote helper" tool to allow Git to
> seamlessly import Subversion history as part of a Google Summer of
> Code project. I later figured that the tool would be useful to both
> SVN server admins to generate backups and other VCS developers; my
> current plan is to get it merged it into git.git, and then (if it's a
> good idea) get it merged into the Subversion tree and remove it from
> git.git. As an added bonus, it currently performs significantly better
> than `svnsync` because it doesn't need to touch the filesystem or
> apply any deltas.
> 
> So far, I've only been developing it as an independent project, and I
> posted a series reporting my progress last night [1]. The series
> includes a nice validation script. I'd really appreciate it if you
> could review, test, and deem the series fit for the Subversion trunk.
> 
> So, is it a good idea to merge it into the Subversion tree? If it is,
> I can re-license the project, generate patches against the Subversion
> tree excluding debug_editor.c, LICENSE, validation.sh and other noise
> to get the merge process started. I can probably even get it to plug
> into `svnsync` as another editor.

I see value in adding this feature to Subversion.
Generating a dump file from a remote repository certainly has a
couple of interesting use cases.

I'd prefer adding this as a new command, such as "svnrdump" ("remote dump").

Obviously, we'll need a patch in proper shape. That means it's against
our trunk (it doesn't matter whether you produce diff output with git or
with svn), and that the coding style (indentation etc.) matches the
conventions used in svn, and that the tool behaves like the other CLI
tools we have regarding option parsing, help output, etc.

Also, please don't send one patch per file, but send the entire thing
in one patch.

A set of unit tests for the new command within our regression test
suite would be nice (similar to svnlook_tests.py and svnsync_tests.py).
These could be submitted in a separate patch.

Unless you hear objections during the next few days, I'd suggest you
start working on a patch aimed at our trunk (instead of at git).

I'm not sure how GSOC handles cases like this though. Please make sure
your mentor is aware of patches you send here, and is willing to work
with you even if you direct them at Subversion rather than git.
I don't expect a problem (being a gsoc mentor myself this year and last
year I have had good experience with the program and the people who
run it), but you should certainly make sure Google is still willing to
support you even if you technically switch projects while summer of code
is running.

Thanks,
Stefan

Re: Proposal for a new exporter tool

[Daniel Shahaf <d....@daniel.shahaf.name>]

Ramkumar Ramachandra wrote on Wed, 7 Jul 2010 at 19:45 +0200:
> > How about adding a pre-* hook so that an administrator could disable
> > this?  There might be some administrators that would want to disable
> > this.

I'm not convinced we should have this; but if we do, shouldn't it be
a pre-replay hook (so it captures svnsync attempts as well)?

Re: Proposal for a new exporter tool

[Ramkumar Ramachandra <ar...@gmail.com>]

Hi Mark,

Mark Phippard writes:
> On Wed, Jul 7, 2010 at 10:47 AM, Ramkumar Ramachandra
> <ar...@gmail.com> wrote:
> 
> > I have recently developed a new svn_editor_t editor for Subversion
> > that uses the replay API to generate a dumpfile v3 over the network
> > on-the-fly (without touching the filesystem except for some temporary
> > files). I initially started building the tool to merge it into the
> > git.git tree and link it to a "remote helper" tool to allow Git to
> > seamlessly import Subversion history as part of a Google Summer of
> > Code project. I later figured that the tool would be useful to both
> > SVN server admins to generate backups and other VCS developers; my
> > current plan is to get it merged it into git.git, and then (if it's a
> > good idea) get it merged into the Subversion tree and remove it from
> > git.git. As an added bonus, it currently performs significantly better
> > than `svnsync` because it doesn't need to touch the filesystem or
> > apply any deltas.
> 
> Since it uses the RA layers, how does this interact with authz rules?
> Does it respect any authz that has been established, just as svnsync
> does?

Thanks for the heads up. I'm looking at authz.c to figure out how to
do this.

> How about adding a pre-* hook so that an administrator could disable
> this?  There might be some administrators that would want to disable
> this.

Like you suggested, I'll handle this after the initial merge.

> Other than that, it sounds cool and like it would be a nice way to
> setup a replica of an existing repository that is subsequently
> maintained by svnsync.  IOW, rather than use svnsync to grab those
> first 100,000 revisions, you could use this tool and then use svnsync
> going forward to sync future commits as they happen.

I suppose that's possible- you have to first `svnadmin load` the
dumpfile and then use `svnsync` to go forward. I don't see why you
need to do that though: you can just specify the starting revision
from where you left off in the previous fetch in svndumpr. A useful
feature: parse the dumpfile to find out how many revisions have
already been dumped and set the starting revision number in svndumpr
accordingly.

-- Ram

Re: Proposal for a new exporter tool

[Ramkumar Ramachandra <ar...@gmail.com>]

Hi Mark,

Mark Phippard writes:
> On Wed, Jul 7, 2010 at 10:47 AM, Ramkumar Ramachandra
> <ar...@gmail.com> wrote:
> 
> > I have recently developed a new svn_editor_t editor for Subversion
> > that uses the replay API to generate a dumpfile v3 over the network
> > on-the-fly (without touching the filesystem except for some temporary
> > files). I initially started building the tool to merge it into the
> > git.git tree and link it to a "remote helper" tool to allow Git to
> > seamlessly import Subversion history as part of a Google Summer of
> > Code project. I later figured that the tool would be useful to both
> > SVN server admins to generate backups and other VCS developers; my
> > current plan is to get it merged it into git.git, and then (if it's a
> > good idea) get it merged into the Subversion tree and remove it from
> > git.git. As an added bonus, it currently performs significantly better
> > than `svnsync` because it doesn't need to touch the filesystem or
> > apply any deltas.
> 
> Since it uses the RA layers, how does this interact with authz rules?
> Does it respect any authz that has been established, just as svnsync
> does?

Actually, I realized that svn_delta_editor_t comes with absent_file
and absent_directory to handle exactly this. By default, they're
no-ops, so I don't think I need to write in any extra code to handle
authz.

> How about adding a pre-* hook so that an administrator could disable
> this?  There might be some administrators that would want to disable
> this.

I thought about this, and I'm not too much in favor of this either
(see Daniel's comment): should we really give the admins an option to
block replay? If replay is disabled on a server, people can still
extract the revision history the brute-force way: fetch the text and
props for each revision by hand, and this is not pretty.

-- Ram

Re: Proposal for a new exporter tool

[Mark Phippard <ma...@gmail.com>]

On Wed, Jul 7, 2010 at 10:47 AM, Ramkumar Ramachandra
<ar...@gmail.com> wrote:

> I have recently developed a new svn_editor_t editor for Subversion
> that uses the replay API to generate a dumpfile v3 over the network
> on-the-fly (without touching the filesystem except for some temporary
> files). I initially started building the tool to merge it into the
> git.git tree and link it to a "remote helper" tool to allow Git to
> seamlessly import Subversion history as part of a Google Summer of
> Code project. I later figured that the tool would be useful to both
> SVN server admins to generate backups and other VCS developers; my
> current plan is to get it merged it into git.git, and then (if it's a
> good idea) get it merged into the Subversion tree and remove it from
> git.git. As an added bonus, it currently performs significantly better
> than `svnsync` because it doesn't need to touch the filesystem or
> apply any deltas.

Since it uses the RA layers, how does this interact with authz rules?
Does it respect any authz that has been established, just as svnsync
does?

How about adding a pre-* hook so that an administrator could disable
this?  There might be some administrators that would want to disable
this.

Other than that, it sounds cool and like it would be a nice way to
setup a replica of an existing repository that is subsequently
maintained by svnsync.  IOW, rather than use svnsync to grab those
first 100,000 revisions, you could use this tool and then use svnsync
going forward to sync future commits as they happen.

-- 
Thanks

Mark Phippard
http://markphip.blogspot.com/