You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by "Eric S. Raymond" <es...@thyrsus.com> on 2009/10/07 01:37:58 UTC
svnsquash has grown new capabilities
Recently I submitted a tool I then called "svnsquash" to this list that
could clean up scar tissue from CVS and old versions of Emacs VC.
Unhappily, I have recently had to enhance this tool considerably.
Last Friday, the hosting site for one of my projects crashed in a rather
permanent-seeming way. As I had to try to reconstruct the project
history from a two-week-old Subversion dump and a git mirror of more
recent commits, the next 72 hours involved a lot of frantic repository
surgery.
Rather than hand-hack, I upgraded my tool each time I had to perform a
new surgical maneuver; there was enough cut-and-try involved that I
saved a lot of time by being able to repeat operations reliably. As a
result, the tool now has the following additional capabilities:
* Set a specified revision property to a specified value on a range of
revisions.
* Remove a specified revision property on a specified range of
revisions.
* Take a file in the format of an svn log report and use its content to set the
log property of each revision mentioned in the file.
Note, these are all operations on *dumpfiles*, not repos. The tool
remains a filter transforming a dumpfile on stdin to a dumpfile on
stdout. This has advantages I don't think I need to belabor here.
The tool's new name, after all that surgery, is "svncutter". I am
probably going to add the ability to excise a specified set of
revisions (extending the existing ability to select a range)
within the next few hours.
My problem is this. The script is no longer a casual hack with
just enough substance to belong in a contrib directory - I'm probably
going to write some actual unit tests for it tonight. But it's also not
large enough, or separable enough from Subversion-as-it-is, that I
feel it makes sense to ship it as a standalone project.
Your advice is solicited.
--
<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>
The end move in politics is always to pick up a gun.
-- R. Buckminster Fuller
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2404350
Re: svnsquash has grown new capabilities
Posted by Karl Fogel <kf...@red-bean.com>.
"Eric S. Raymond" <es...@thyrsus.com> writes:
> Eric S. Raymond <es...@thyrsus.com>:
>> Noted. I'll resubmit when I have things polished to my satisfaction.
>
> What do you guys like as a format for manual-page masters? I've
> already written pretty much all the content I'd need for one in svncutter's
> embedded help.
I don't think there's any standard for contrib/ stuff. Do what you
think works best for the tool (embedded help is fine), and then maybe
have a one-line man page saying "Run it with --help for usage" :-).
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2404687
Re: svnsquash has grown new capabilities
Posted by "Eric S. Raymond" <es...@thyrsus.com>.
Eric S. Raymond <es...@thyrsus.com>:
> Noted. I'll resubmit when I have things polished to my satisfaction.
What do you guys like as a format for manual-page masters? I've
already written pretty much all the content I'd need for one in svncutter's
embedded help.
--
<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2404667
Re: svnsquash has grown new capabilities
Posted by "Eric S. Raymond" <es...@thyrsus.com>.
Blair Zajac <bl...@orcaware.com>:
> The one I've needed is when you're pulling a project out of a larger
> repository into a new smaller one, so you want to the project up one
> or more directory levels. For example, project foobar has a
> directory foobar in the repository
>
> foobar/
> trunk/
> tags/
> branches/
>
> and you want a new foobar repos with just trunk, tags and branches at the top
> level. So svndumpfilter works for pulling foobar, but not for moving it up.
Agreed, that one seems worth doing. And not at all difficult. I'll probably
implement it as "svncutter lift" within the next few hours.
--
<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2404686
Re: svnsquash has grown new capabilities
Posted by "Eric S. Raymond" <es...@thyrsus.com>.
Martin Furter <mf...@rola.ch>:
> The 'merge' command of svndumptool can do that, and it's perfectly fine to
> merge only one dumpfile.
>
> svndumptool.py merge -i in.dmp -x foobar -r foobar/trunk trunk ... -o out.dmp
I could not ask for a nore perfect illustration of my point in reply
to David Glasser. I had no idea this tool existed - it is exactly the
sort of low-overhead single-file project he was proposing I make
svncutter.
Had I known, I probably would have enhanced a copy to do what I wanted and
given the changes back to the maintainer. As it is, I shall now write him
and suggest that these projects should merge.
--
<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2404732
Re: svnsquash has grown new capabilities
Posted by "Eric S. Raymond" <es...@thyrsus.com>.
Martin Furter <mf...@rola.ch>:
> Improving the [svndumptool.py] help output is also on the todo list...
I can be some help with that. I'm a native English-speaker, of course,
and I don't mind writing documentation.
--
<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2405893
Re: svnsquash has grown new capabilities
Posted by Martin Furter <mf...@rola.ch>.
On Wed, 7 Oct 2009, Blair Zajac wrote:
> Martin Furter wrote:
>> On Wed, 7 Oct 2009, Blair Zajac wrote:
>>
>>> The one I've needed is when you're pulling a project out of a larger
>>> repository
>>> into a new smaller one, so you want to the project up one or more
>>> directory levels.
>>>
>>> For example, project foobar has a directory foobar in the repository
>>>
>>> foobar/
>>> trunk/
>>> tags/
>>> branches/
>>>
>>> and you want a new foobar repos with just trunk, tags and branches at the
>>> top
>>> level. So svndumpfilter works for pulling foobar, but not for moving it
>>> up.
>>
>> The 'merge' command of svndumptool can do that, and it's perfectly fine to
>> merge only one dumpfile.
>>
>> svndumptool.py merge -i in.dmp -x foobar -r foobar/trunk trunk ... -o
>> out.dmp
>
> I did install svndumptool before editing my dump file by hand to see if it
> had this functionality but didn't see an appropriate subcommand.
>
> Maybe there could be an additional subcommand for doing this? Or perhaps
> 'svndumptool help' would show it.
Yes, merge should be split into multiple commands. Implementing the squash
command of svncutter could already use some code which is hidden in merge.
Improving the help output is also on the todo list...
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2405874
Re: svnsquash has grown new capabilities
Posted by Blair Zajac <bl...@orcaware.com>.
Martin Furter wrote:
> On Wed, 7 Oct 2009, Blair Zajac wrote:
>
>> The one I've needed is when you're pulling a project out of a larger repository
>> into a new smaller one, so you want to the project up one or more directory levels.
>>
>> For example, project foobar has a directory foobar in the repository
>>
>> foobar/
>> trunk/
>> tags/
>> branches/
>>
>> and you want a new foobar repos with just trunk, tags and branches at the top
>> level. So svndumpfilter works for pulling foobar, but not for moving it up.
>
> The 'merge' command of svndumptool can do that, and it's perfectly fine to
> merge only one dumpfile.
>
> svndumptool.py merge -i in.dmp -x foobar -r foobar/trunk trunk ... -o out.dmp
I did install svndumptool before editing my dump file by hand to see if it had
this functionality but didn't see an appropriate subcommand.
Maybe there could be an additional subcommand for doing this? Or perhaps
'svndumptool help' would show it.
Regards,
Blair
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2404757
Re: svnsquash has grown new capabilities
Posted by Martin Furter <mf...@rola.ch>.
On Wed, 7 Oct 2009, Blair Zajac wrote:
> The one I've needed is when you're pulling a project out of a larger repository
> into a new smaller one, so you want to the project up one or more directory levels.
>
> For example, project foobar has a directory foobar in the repository
>
> foobar/
> trunk/
> tags/
> branches/
>
> and you want a new foobar repos with just trunk, tags and branches at the top
> level. So svndumpfilter works for pulling foobar, but not for moving it up.
The 'merge' command of svndumptool can do that, and it's perfectly fine to
merge only one dumpfile.
svndumptool.py merge -i in.dmp -x foobar -r foobar/trunk trunk ... -o out.dmp
HTH
Martin
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2404720
Re: svnsquash has grown new capabilities
Posted by Blair Zajac <bl...@orcaware.com>.
Eric S. Raymond wrote:
> Mark Phippard <ma...@gmail.com>:
>>> Being in contrib/ doesn't mean "casual hack". It just means "maintained
>>> by a contributor", i.e., the Subversion project is distributing it along
>>> with Subversion but not taking any responsibility for it. So IMHO
>>> leaving it in contrib is the best course. Giving it unit tests is fine.
>> If there are several files, then it is probably worthy of a
>> sub-directory in contrib as we have done for other tools. Something
>> like contrib/server-side/svncutter
>>
>> Then a README and the tests etc. can all go into that folder.
>
> Noted. I'll resubmit when I have things polished to my satisfaction.
>
> Revision ranges for the -r option can now be comma-separated lists of
> ranges, with the obvious interpretation.
>
> I've added the abiliity to rename a property over the specified
> revision set. This is useful because svn commit helpfully won't
> twiddle svn:author and svn:date - at one point in my disaster
> recovery I was applying git diffs to a repo and had to stash these
> properties in git:auther and git:date, renaming them later with a
> silly little filter script. The proprename subcommand replaces that
> silly little filter script.
>
> I just got the 'excise' option for unconditionally removing specified
> commits during a squash working - I wanted this to be able to remove
> certain cvs2svn conversion artifacts.
>
> Another previously implemented subcommand I forgot to mention is "log"
> - run it and get the same output as if you had done svn log run on a
> repo created from the dump. I used "svncutter log" followed by an Emacs
> session followed by "svncutter setlog" to fix up all the annoying typos
> in my comment history.
>
> The current subcommand set (squash, select, propdel, propset, proprename,
> log, setlog) is all I had planned for the tool. I'll take requests, however,
> if any of you can think of a frequently requested dumpfile-surgery operarion
> that I've missed.
The one I've needed is when you're pulling a project out of a larger repository
into a new smaller one, so you want to the project up one or more directory levels.
For example, project foobar has a directory foobar in the repository
foobar/
trunk/
tags/
branches/
and you want a new foobar repos with just trunk, tags and branches at the top
level. So svndumpfilter works for pulling foobar, but not for moving it up.
Regards,
Blair
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2404663
Re: svnsquash has grown new capabilities
Posted by "Eric S. Raymond" <es...@thyrsus.com>.
Mark Phippard <ma...@gmail.com>:
> > Being in contrib/ doesn't mean "casual hack". It just means "maintained
> > by a contributor", i.e., the Subversion project is distributing it along
> > with Subversion but not taking any responsibility for it. So IMHO
> > leaving it in contrib is the best course. Giving it unit tests is fine.
>
> If there are several files, then it is probably worthy of a
> sub-directory in contrib as we have done for other tools. Something
> like contrib/server-side/svncutter
>
> Then a README and the tests etc. can all go into that folder.
Noted. I'll resubmit when I have things polished to my satisfaction.
Revision ranges for the -r option can now be comma-separated lists of
ranges, with the obvious interpretation.
I've added the abiliity to rename a property over the specified
revision set. This is useful because svn commit helpfully won't
twiddle svn:author and svn:date - at one point in my disaster
recovery I was applying git diffs to a repo and had to stash these
properties in git:auther and git:date, renaming them later with a
silly little filter script. The proprename subcommand replaces that
silly little filter script.
I just got the 'excise' option for unconditionally removing specified
commits during a squash working - I wanted this to be able to remove
certain cvs2svn conversion artifacts.
Another previously implemented subcommand I forgot to mention is "log"
- run it and get the same output as if you had done svn log run on a
repo created from the dump. I used "svncutter log" followed by an Emacs
session followed by "svncutter setlog" to fix up all the annoying typos
in my comment history.
The current subcommand set (squash, select, propdel, propset, proprename,
log, setlog) is all I had planned for the tool. I'll take requests, however,
if any of you can think of a frequently requested dumpfile-surgery operarion
that I've missed.
--
<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2404661
Re: svnsquash has grown new capabilities
Posted by Mark Phippard <ma...@gmail.com>.
On Wed, Oct 7, 2009 at 3:21 PM, Karl Fogel <kf...@red-bean.com> wrote:
> "Eric S. Raymond" <es...@thyrsus.com> writes:
>> My problem is this. The script is no longer a casual hack with
>> just enough substance to belong in a contrib directory - I'm probably
>> going to write some actual unit tests for it tonight. But it's also not
>> large enough, or separable enough from Subversion-as-it-is, that I
>> feel it makes sense to ship it as a standalone project.
>>
>> Your advice is solicited.
>
> Being in contrib/ doesn't mean "casual hack". It just means "maintained
> by a contributor", i.e., the Subversion project is distributing it along
> with Subversion but not taking any responsibility for it. So IMHO
> leaving it in contrib is the best course. Giving it unit tests is fine.
If there are several files, then it is probably worthy of a
sub-directory in contrib as we have done for other tools. Something
like contrib/server-side/svncutter
Then a README and the tests etc. can all go into that folder.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2404647
Re: svnsquash has grown new capabilities
Posted by Karl Fogel <kf...@red-bean.com>.
"Eric S. Raymond" <es...@thyrsus.com> writes:
> My problem is this. The script is no longer a casual hack with
> just enough substance to belong in a contrib directory - I'm probably
> going to write some actual unit tests for it tonight. But it's also not
> large enough, or separable enough from Subversion-as-it-is, that I
> feel it makes sense to ship it as a standalone project.
>
> Your advice is solicited.
Being in contrib/ doesn't mean "casual hack". It just means "maintained
by a contributor", i.e., the Subversion project is distributing it along
with Subversion but not taking any responsibility for it. So IMHO
leaving it in contrib is the best course. Giving it unit tests is fine.
-Karl
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2404642
Re: svnsquash has grown new capabilities
Posted by Blair Zajac <bl...@orcaware.com>.
Eric S. Raymond wrote:
> David Glasser <gl...@davidglasser.net>:
>> My advice: what do you mean when you say that it does not make sense
>> to ship it as a standalone project?
>
> It's so tied to the internals of Subversion that I think I'd prefer to
> have it co-located with Subversion. And there's some risk that if I create
> it as a standalone project, people simply won't *find* it. One corollary of
> projects-are-cheap is that learning what's relevant to what you want to do
> can be difficult. The problems of superabundance...
>
> Yes, Google searches mitigate the problem, but they don't entirely
> solve it. The tradeoff I'm choosing, consciously, is to go through
> your bureaucracy in order to make the tool more easily discoverable to
> those most likely to need it. In this *particular* case the choice is
> esier because I'm pretty sure that if I were to come to this dev list
> in the future and said "Hey, I need commit access now to tweak
> svncutter" I'd get it without fuss.
>
> That said, I mostly agree with your more general point. Project creation abd
> hosting are cheap. We should behave more like that's true than we often do.
As one of the svn developers, I'm more of the mind of Eric than David here, so
don't agree with the Subversion developers that want to see contrib phased out.
I do think there's value in having one central place to store smaller tools than
having to go look for them elsewhere. It is nice to go to contrib/ or tools/ to
look and see if there's a tool you need or could use. If each of the tools were
a separate package, then as a newer Subversion developer/administrator, you may
not even find out that a particular tool is useful, (say case_insensitive.py to
protect yourself from case issues on Windows and Mac clients).
Blair
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2405198
Re: svnsquash has grown new capabilities
Posted by "Eric S. Raymond" <es...@thyrsus.com>.
David Glasser <gl...@davidglasser.net>:
> My advice: what do you mean when you say that it does not make sense
> to ship it as a standalone project?
It's so tied to the internals of Subversion that I think I'd prefer to
have it co-located with Subversion. And there's some risk that if I create
it as a standalone project, people simply won't *find* it. One corollary of
projects-are-cheap is that learning what's relevant to what you want to do
can be difficult. The problems of superabundance...
Yes, Google searches mitigate the problem, but they don't entirely
solve it. The tradeoff I'm choosing, consciously, is to go through
your bureaucracy in order to make the tool more easily discoverable to
those most likely to need it. In this *particular* case the choice is
esier because I'm pretty sure that if I were to come to this dev list
in the future and said "Hey, I need commit access now to tweak
svncutter" I'd get it without fuss.
That said, I mostly agree with your more general point. Project creation abd
hosting are cheap. We should behave more like that's true than we often do.
I was thinking of posting something semi-relevant here about the state
of modern hosting systems. Now I'm sure I will. I think you'll find it
interesting.
--
<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2404710
Re: svnsquash has grown new capabilities
Posted by David Glasser <gl...@davidglasser.net>.
On Tue, Oct 6, 2009 at 6:37 PM, Eric S. Raymond <es...@thyrsus.com> wrote:
> My problem is this. The script is no longer a casual hack with
> just enough substance to belong in a contrib directory - I'm probably
> going to write some actual unit tests for it tonight. But it's also not
> large enough, or separable enough from Subversion-as-it-is, that I
> feel it makes sense to ship it as a standalone project.
>
> Your advice is solicited.
My advice: what do you mean when you say that it does not make sense
to ship it as a standalone project?
In my opinion, the biggest success of Subversion is that it has made
creating and hosting new projects a much lower-overhead operation than
it was in the last millennium. (I mean this both through Subversion
itself, and through the host of other open source version control
projects in this VC Renaissance that Subversion ushered in.) Have a
little program that you want to share with the world? It takes just a
few clicks to set up a new project on Google Code, Github, or a host
of many other project hosting sites.
With your own project, you can decide to give commit access to other
interested parties without having to go through the Subversion
committers. With most of these project hosting sites, you can put up
your own releases and downloads on your own schedule without having to
deal with "oh, we haven't fixed all our release blockers so our
tarball can't go out yet".
Frankly, "every software project has to be some enormous,
high-overhead thing" is exactly the sort of attitude that comes from
clunky, difficult tools.
In my opinion, the contrib directory of Subversion is outdated and
should be phased out, and just converted to a list of links on a wiki
or something. Other Subversion developers agree with me on this, and
we may change our policy officially in the future.
That said, we haven't changed our policy yet, and if you really
believe that maintaining your project inside the bureaucracy of our
project actually makes sense for you, I'm pretty sure we'll say yes.
But is that actually more beneficial than taking 10 minutes to set up
a one-file project on a hosting site, and then never having to worry
about our project's particular quirks and details again?
--dave
--
glasser@davidglasser.net | langtonlabs.org | flickr.com/photos/glasser/
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=2404700