You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Nik Clayton <ni...@ngo.org.uk> on 2006/09/17 15:40:06 UTC
Issues 516: "svn obliterate"
Hi,
Is anyone actively working on implementing an API to completely remove a
file from a repository, as described in:
http://subversion.tigris.org/issues/show_bug.cgi?id=516
?
I'm looking at what would be involved in migrating FreeBSD from CVS to
Subversion, and we think we need this functionality.
As well as the traditional "Someone's committed a core file by accident"
use case that's covered in that issue, we have another use case that
doesn't seem to have been discussed (at least, a quick search didn't
turn it up).
This is where a committer inadvertently commits code that they are not
licensed to commit. This might be a binary blob from a third party
vendor that's not supposed to be distributed, or it might be, as has
happened to FreeBSD, been code being developed for a commercial third
party, that will eventually be open sourced, but that is not supposed to
be open sourced yet.
That's happened to FreeBSD. Some emergency CVS repo surgery meant that
the offending code was removed from the master repo, and all the mirrors
shortly caught up.
I don't (yet) have a solution that will work with Subversion. I'm aware
that it's possible to dump the repo from just before and just after the
commit, load the first dump, manufacture a dummy commit, and then load
the second dump.
However, I don't yet know how feasible that is. First, the dump/load
cycle for a large repository is lengthy -- doing this to a hypothetical
FreeBSD SVN repo, which will contain 10+ years of imported CVS history
is likely to be prohibitively so. Second, I haven't tested what the
effects of this procedure would be on any one who has mirrored the repo
(using svnsync) or has a working copy that contains the offending item.
Alternative approaches, such as restore from a backup prior to the
commit, and replay the remaining commits are also possible, but again,
I've yet to investigate what sort of effect this would have on mirrors
that are svnsync(1)'ing the repo.
Hence our interest in supported command/API that does this.
N
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Issues 516: "svn obliterate"
Posted by Karl Fogel <kf...@google.com>.
Grant Rettke <gr...@acm.org> writes:
> Subversion's features are really driven by its customers. Right now,
> there are two groups customers: Collab.net's customers and the open source
> customers of Subversion (GCC seems to be a big one).
There are many, many commercial users of Subversion, some are
CollabNet's customers and some aren't. See
http://www.vsoft-tech.com.au/Default.aspx?tabid=77&EntryID=190
for some loose stats (I got that from Ted Dennison's posts on the
users@ list recently).
> If commercial users really want 'svn obliterate', they would pay for it.
>
> If open source users really want 'svn obliterate', it would come about
> probably
> as the result of a lawsuit or something.
>
> Based on both the bugzilla entry and emails in this group, it sounds
> like 'svn obliterate' is non-trivial both from an implementation and
> use-case perspective.
I agree. The problem with obliterate is that it's hard to spec, and
it may be hard to implement some aspects of it as well. My estimate
is that if someone could put up appx $10k (US) to hire a Subversion
developer to start the discussion, follow it through, and implement
it, then 'svn obliterate' will happen. Without that, it will probably
still happen eventually, just not nearly as soon.
-Karl
> Quoting Nik Clayton <ni...@ngo.org.uk>:
>
>> Hi,
>>
>> Is anyone actively working on implementing an API to completely
>> remove a file from a repository, as described in:
>>
>> http://subversion.tigris.org/issues/show_bug.cgi?id=516
>>
>> ?
>>
>> I'm looking at what would be involved in migrating FreeBSD from CVS
>> to Subversion, and we think we need this functionality.
>>
>> As well as the traditional "Someone's committed a core file by
>> accident" use case that's covered in that issue, we have another use
>> case that doesn't seem to have been discussed (at least, a quick
>> search didn't turn it up).
>>
>> This is where a committer inadvertently commits code that they are
>> not licensed to commit. This might be a binary blob from a third
>> party vendor that's not supposed to be distributed, or it might be,
>> as has happened to FreeBSD, been code being developed for a
>> commercial third party, that will eventually be open sourced, but
>> that is not supposed to be open sourced yet.
>>
>> That's happened to FreeBSD. Some emergency CVS repo surgery meant
>> that the offending code was removed from the master repo, and all
>> the mirrors shortly caught up.
>>
>> I don't (yet) have a solution that will work with Subversion. I'm
>> aware that it's possible to dump the repo from just before and just
>> after the commit, load the first dump, manufacture a dummy commit,
>> and then load the second dump.
>>
>> However, I don't yet know how feasible that is. First, the
>> dump/load cycle for a large repository is lengthy -- doing this to a
>> hypothetical FreeBSD SVN repo, which will contain 10+ years of
>> imported CVS history is likely to be prohibitively so. Second, I
>> haven't tested what the effects of this procedure would be on any
>> one who has mirrored the repo (using svnsync) or has a working copy
>> that contains the offending item.
>>
>> Alternative approaches, such as restore from a backup prior to the
>> commit, and replay the remaining commits are also possible, but
>> again, I've yet to investigate what sort of effect this would have
>> on mirrors that are svnsync(1)'ing the repo.
>>
>> Hence our interest in supported command/API that does this.
>>
>> N
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
>> For additional commands, e-mail: dev-help@subversion.tigris.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: dev-help@subversion.tigris.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Issues 516: "svn obliterate"
Posted by Karl Fogel <kf...@google.com>.
Nik Clayton <ni...@ngo.org.uk> writes:
> Oh, and I hope that this isn't coming across as a demand that this
> feature be implemented. I know it's been requested before (hence the
> issue tracker reference), but all of the use cases that I've seen have
> referred to inadvertently committing large files. There's not been a
> reference to inadvertently committing files which might get you in
> legal trouble.
http://subversion.tigris.org/issues/show_bug.cgi?id=516#desc17
talks about this, actually.
Best,
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Issues 516: "svn obliterate"
Posted by ph...@bmlv.gv.at.
May I try a quick summary of what I heard here, an a few conclusions?
Nik Clayton <ni...@ngo.org.uk> wrote:
I can envisage a solution that involves making regular hot copies of the
repo (say, every hour), and should a problem occur, revert back to the
previous hour's backup, and replay all the commits except the offending
one since then.
That sounds a bit like versioning the repository, which could be an easy task if
done to a fsfs repository with a fsfs repository.
In case of a problem the repository is reverted back to the old revision (update
-rOLD), the backend-repository gets its revision number lowered and the
revisions above deleted.
If later commits would be nice to retain, they should be exported from the
backend-repository before changing HEAD.
As to the technical aspect of "svn obliterate", AFAIK the delta mechanism stores
the blobs needed to construct any version.
What happens if the relevant blobs are simply zeroed-out, so that these parts
contain only NUL-bytes? Of course, then we're back to the problem with
dependant revisions ...
Of course, that all won't help as long as someone has updated on the right time
- the .svn directories will hold the "bad" version for a potentially long time
...
That brings us to a new aspect: Every "svn obliterate" *must* (IMO) create a new
revision which marks all obliterated files as changed, to get the clients to
fetch a "clean" state. Whether that state is deleted, 0byte length, or
whatever, doesn't matter.
Regards,
Phil
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Issues 516: "svn obliterate"
Posted by Garrett Rooney <ro...@electricjellyfish.net>.
On 9/18/06, Daniel Rall <dl...@collab.net> wrote:
> On Mon, 18 Sep 2006, Nik Clayton wrote:
>
> > Jacob Atzen wrote:
> > >On Mon, Sep 18, 2006 at 09:40:03PM +0100, Nik Clayton wrote:
> > >>I also don't know what happens to svnsync'd copies of repos when you do
> > >>this. That's something else for me to play with.
> > >
> > >Have you considered using rsync for mirroring the repository? I'm not
> > >sure if this is even feasible with your setup, but it could be a
> > >solution to the mirroring issue. More info is available at:
> > >
> > >http://snarfed.org/space/backup+subversion+fsfs+with+rsync
> > >
> > >Beware that this afaik only works with the FSFS backend.
> >
> > Yeah, looking at this is on my todo list -- if svnsync can't cope with
> > repos that have gone through a dump/load cycle then this is my backup plan.
>
> Using dump/load to emulate 'obliterate' for a master repository which
> is the source for a mirrored repository may or may not hork up any
> subsequent svnsync (I think "not" is more likely, if you avoid
> modifying the revnums to account for the filtered revision(s)), but it
> definitely won't propogate the 'obliterate'.
>
Yeah, as long as the revnums match it'll still sync, but there won't
be any propogation of the removed revs, since that's outside the
understanding of the tools in question.
-garrett
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Issues 516: "svn obliterate"
Posted by Daniel Rall <dl...@collab.net>.
On Mon, 18 Sep 2006, Nik Clayton wrote:
> Jacob Atzen wrote:
> >On Mon, Sep 18, 2006 at 09:40:03PM +0100, Nik Clayton wrote:
> >>I also don't know what happens to svnsync'd copies of repos when you do
> >>this. That's something else for me to play with.
> >
> >Have you considered using rsync for mirroring the repository? I'm not
> >sure if this is even feasible with your setup, but it could be a
> >solution to the mirroring issue. More info is available at:
> >
> >http://snarfed.org/space/backup+subversion+fsfs+with+rsync
> >
> >Beware that this afaik only works with the FSFS backend.
>
> Yeah, looking at this is on my todo list -- if svnsync can't cope with
> repos that have gone through a dump/load cycle then this is my backup plan.
Using dump/load to emulate 'obliterate' for a master repository which
is the source for a mirrored repository may or may not hork up any
subsequent svnsync (I think "not" is more likely, if you avoid
modifying the revnums to account for the filtered revision(s)), but it
definitely won't propogate the 'obliterate'.
That said, you're going to have similar problems with your change log
emails, and any web-based email archives of them (which you may or may
not control).
This "oops, I committed proprietary IP where I shouldn't have" problem
definitely isn't unique to your situation. It's a tough one to deal
with. :)
Re: Issues 516: "svn obliterate"
Posted by Nik Clayton <ni...@ngo.org.uk>.
Jacob Atzen wrote:
> On Mon, Sep 18, 2006 at 09:40:03PM +0100, Nik Clayton wrote:
>> I also don't know what happens to svnsync'd copies of repos when you do
>> this. That's something else for me to play with.
>
> Have you considered using rsync for mirroring the repository? I'm not
> sure if this is even feasible with your setup, but it could be a
> solution to the mirroring issue. More info is available at:
>
> http://snarfed.org/space/backup+subversion+fsfs+with+rsync
>
> Beware that this afaik only works with the FSFS backend.
Yeah, looking at this is on my todo list -- if svnsync can't cope with
repos that have gone through a dump/load cycle then this is my backup plan.
N
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Issues 516: "svn obliterate"
Posted by Jacob Atzen <ja...@jacobatzen.dk>.
On Mon, Sep 18, 2006 at 09:40:03PM +0100, Nik Clayton wrote:
> I also don't know what happens to svnsync'd copies of repos when you do
> this. That's something else for me to play with.
Have you considered using rsync for mirroring the repository? I'm not
sure if this is even feasible with your setup, but it could be a
solution to the mirroring issue. More info is available at:
http://snarfed.org/space/backup+subversion+fsfs+with+rsync
Beware that this afaik only works with the FSFS backend.
--
Cheers,
- Jacob Atzen
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Issues 516: "svn obliterate"
Posted by Nik Clayton <ni...@ngo.org.uk>.
Hi Brian, long time no see.
Brian Behlendorf wrote:
> But I have not heard of it being the sole reason why one of our
> customers or prospects decided against the move from CVS to SVN, nor
> from any other SCM tool. That's why CollabNet's not focused on it at
> the moment, and instead on things like merge tracking and overall
> performance & quality.
Yes. To be clear, this is not necessarily a show stopper for FreeBSD.
At the moment I'm gathering requirements that need to be considered in
migrating to any new VCS, and looking at how Subversion (in my case,
others are looking at other VCSs) might address these requirements.
FWIW, you can see the current list at:
http://wikitest.freebsd.org/VersionControl
Oh, and I hope that this isn't coming across as a demand that this
feature be implemented. I know it's been requested before (hence the
issue tracker reference), but all of the use cases that I've seen have
referred to inadvertently committing large files. There's not been a
reference to inadvertently committing files which might get you in legal
trouble.
> It does have a sort of stuffing-the-genie-back-in-the-bottle feel to it.
> Commits usually trigger emails, and you can't recall those emails.
> Backups kept by a mirror will have that content.
All true. And I think the thing to focus on here is 'best effort'.
When this happened in the past to the FreeBSD project the lawyers were
happy that we'd made best efforts to remove the offending code, even if
we couldn't guarantee that it had been completely removed.
> Nik, I don't mean to discount your desire for this feature, but it might
> be useful to ask whether there are ways to mitigate the need for it.
Indeed. The dump/load cycle is one that I need to investigate more, but
my previous experiences of that cycle show it taking much longer than
we'd probably like.
I can envisage a solution that involves making regular hot copies of the
repo (say, every hour), and should a problem occur, revert back to the
previous hour's backup, and replay all the commits except the offending
one since then.
Of course, that trades disk space for time, and I don't know yet how
large the FreeBSD repo would end up. Perhaps large enough that only a
24 hour backup schedule is appropriate.
I also don't know what happens to svnsync'd copies of repos when you do
this. That's something else for me to play with.
> One thing we suggest to customers who note this and want the same thing
> is to encourage them to use lots of small individual repositories based
> on modules/projects/whatever rather than one single big one. That way,
> needing to do a dump/svnfilter/load cycle can be much shorter, and you
> don't need to take the repo down for every other team,
I don't think that's really a starter for us. One of the potential
benefits is going to be bringing everything back under one repo, and in
particular, putting all the FreeBSD long term development branches,
which currently reside in a Perforce repo, back in to the same repo as
holds the HEAD, so that merges from the development branches can retain
change history.
> and svn:externals helps make it mostly seamless for the average end user.
> The downside is
> a lack of transactional consistancy for changes that span repositories,
> but such changes suggest a lack of proper modularity anyways. It
> doesn't answer the what-about-mirrors question either.
Yes -- I think mirrors are going to mean that svn:externals are a
non-starter for FreeBSD, at least until there's support for 'relative'
externals.
> It's also not clear that the story
> would be any different for any other VC system, except those whose
> repository data model looks like CVS's - a model that prevents
> transactions, versioning of directories, efficient tagging, etc etc.
Also true -- this requirement only surfaced during a face to face
discussion at the weekend, and I don't believe the Mercurial/Hg and Git
proponents have had a chance to respond to it yet.
N
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Issues 516: "svn obliterate"
Posted by Vlad Skvortsov <vs...@73rus.com>.
Garrett Rooney wrote:
>>
>> I would like to bring up slightly different topic here, though.
>> Dump/filter/load cycle can help with getting rid of specific commit, but
>> another issue here is references. Commit log messages often contain
>> references to prior commits - in form of "merged r1234:1239 from /a/b/c"
>> or "follow-up to r1234", etc. It's invaluable piece of information for
>> any nontrivial project and in my opinion it would be great if the
>> proposed procedure was able to keep revision numbering intact (just
>> "blacking out" specific commits).
>
>
> That can be done today (and would be in such a situation), simply
> insert empty revisions in place of dropped ones. I believe
> svndumpfilter already has support for such a thing. Yep, the default
> behavior for svndumpfilter exclude is to leave empty revs, you have to
> use --drop-empty-revs to make it leave out a rev even if there are no
> changes left in it.
Oh, thanks for the pointer - I have overlooked that.
--
Vlad Skvortsov, vss@73rus.com, http://vss.73rus.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Issues 516: "svn obliterate"
Posted by Garrett Rooney <ro...@electricjellyfish.net>.
On 9/27/06, Vlad Skvortsov <vs...@73rus.com> wrote:
> Garrett Rooney wrote:
>
> > <svn.apache.org maintainer hat on>
> > Actually, it would probably be possible to mitigate the pain somewhat
> > by simply turning off access to the portion of the tree in question
> > via authz, and then doing a dump/load in the background to get the
> > data in question out of the tree. It'd be a pain in the neck, but I
> > doubt we'd be stuck taking the whole repos down for the entire
> > process. Note that we have had something like this happen in the
> > past, and we did turn off access to part of the repos during that
> > period of time, although the issue was resolved via some relicensing
> > without the need for a dump/load cycle. Brian is right about the
> > whole "genie back in the bottle" problem though, it's rather difficult
> > to retract data like that once it gets in the open, the only thing
> > that would make us jump through the hoops in question is likely a
> > potential law suit.
> > </svn.apache.org maintainer hat on>
>
> I would like to bring up slightly different topic here, though.
> Dump/filter/load cycle can help with getting rid of specific commit, but
> another issue here is references. Commit log messages often contain
> references to prior commits - in form of "merged r1234:1239 from /a/b/c"
> or "follow-up to r1234", etc. It's invaluable piece of information for
> any nontrivial project and in my opinion it would be great if the
> proposed procedure was able to keep revision numbering intact (just
> "blacking out" specific commits).
That can be done today (and would be in such a situation), simply
insert empty revisions in place of dropped ones. I believe
svndumpfilter already has support for such a thing. Yep, the default
behavior for svndumpfilter exclude is to leave empty revs, you have to
use --drop-empty-revs to make it leave out a rev even if there are no
changes left in it.
-garrett
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Issues 516: "svn obliterate"
Posted by Vlad Skvortsov <vs...@73rus.com>.
Garrett Rooney wrote:
> <svn.apache.org maintainer hat on>
> Actually, it would probably be possible to mitigate the pain somewhat
> by simply turning off access to the portion of the tree in question
> via authz, and then doing a dump/load in the background to get the
> data in question out of the tree. It'd be a pain in the neck, but I
> doubt we'd be stuck taking the whole repos down for the entire
> process. Note that we have had something like this happen in the
> past, and we did turn off access to part of the repos during that
> period of time, although the issue was resolved via some relicensing
> without the need for a dump/load cycle. Brian is right about the
> whole "genie back in the bottle" problem though, it's rather difficult
> to retract data like that once it gets in the open, the only thing
> that would make us jump through the hoops in question is likely a
> potential law suit.
> </svn.apache.org maintainer hat on>
I would like to bring up slightly different topic here, though.
Dump/filter/load cycle can help with getting rid of specific commit, but
another issue here is references. Commit log messages often contain
references to prior commits - in form of "merged r1234:1239 from /a/b/c"
or "follow-up to r1234", etc. It's invaluable piece of information for
any nontrivial project and in my opinion it would be great if the
proposed procedure was able to keep revision numbering intact (just
"blacking out" specific commits).
--
Vlad Skvortsov, vss@73rus.com, http://vss.73rus.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Issues 516: "svn obliterate"
Posted by Garrett Rooney <ro...@electricjellyfish.net>.
On 9/18/06, Brian Behlendorf <br...@collab.net> wrote:
> If this kind of thing happened at Apache while we were using CVS, I think
> we would have responded in the same way as Nik did. If it were to happen
> today, with all of Apache's repositories now under SVN, I think we would
> make the call between either saying "too bad - the company and the
> developer need to figure out the right way to atone for this", or if the
> developer's job really was at risk, taking the repository down for days
> while doing a conversion and then instructing the mirrors to do a clean
> re-mirroring. That's painful but still far less than the pain of sticking
> to CVS would have been. It's also not clear that the story would be any
> different for any other VC system, except those whose repository data
> model looks like CVS's - a model that prevents transactions, versioning of
> directories, efficient tagging, etc etc.
<svn.apache.org maintainer hat on>
Actually, it would probably be possible to mitigate the pain somewhat
by simply turning off access to the portion of the tree in question
via authz, and then doing a dump/load in the background to get the
data in question out of the tree. It'd be a pain in the neck, but I
doubt we'd be stuck taking the whole repos down for the entire
process. Note that we have had something like this happen in the
past, and we did turn off access to part of the repos during that
period of time, although the issue was resolved via some relicensing
without the need for a dump/load cycle. Brian is right about the
whole "genie back in the bottle" problem though, it's rather difficult
to retract data like that once it gets in the open, the only thing
that would make us jump through the hoops in question is likely a
potential law suit.
</svn.apache.org maintainer hat on>
-garrett
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Issues 516: "svn obliterate"
Posted by Brian Behlendorf <br...@collab.net>.
On Mon, 18 Sep 2006, Grant Rettke wrote:
> If commercial users really want 'svn obliterate', they would pay for it.
The architectural quirk in CVS (and I say "quirk" because it does not seem
like it was an intentional design goal for CVS) that allowed Nik to deal
with his late night IP emergency is something that we've heard other
customers note while evaluating a move, and is especially relevant in
non-Open-Source networks involving a lot more complicated scenarios of who
can be trusted with what.
But I have not heard of it being the sole reason why one of our customers
or prospects decided against the move from CVS to SVN, nor from any other
SCM tool. That's why CollabNet's not focused on it at the moment, and
instead on things like merge tracking and overall performance & quality.
It does have a sort of stuffing-the-genie-back-in-the-bottle feel to it.
Commits usually trigger emails, and you can't recall those emails.
Backups kept by a mirror will have that content.
Nik, I don't mean to discount your desire for this feature, but it might
be useful to ask whether there are ways to mitigate the need for it. One
thing we suggest to customers who note this and want the same thing is to
encourage them to use lots of small individual repositories based on
modules/projects/whatever rather than one single big one. That way,
needing to do a dump/svnfilter/load cycle can be much shorter, and you
don't need to take the repo down for every other team, and svn:externals
helps make it mostly seamless for the average end user. The downside is a
lack of transactional consistancy for changes that span repositories, but
such changes suggest a lack of proper modularity anyways. It doesn't
answer the what-about-mirrors question either.
If this kind of thing happened at Apache while we were using CVS, I think
we would have responded in the same way as Nik did. If it were to happen
today, with all of Apache's repositories now under SVN, I think we would
make the call between either saying "too bad - the company and the
developer need to figure out the right way to atone for this", or if the
developer's job really was at risk, taking the repository down for days
while doing a conversion and then instructing the mirrors to do a clean
re-mirroring. That's painful but still far less than the pain of sticking
to CVS would have been. It's also not clear that the story would be any
different for any other VC system, except those whose repository data
model looks like CVS's - a model that prevents transactions, versioning of
directories, efficient tagging, etc etc.
Brian
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Issues 516: "svn obliterate"
Posted by Grant Rettke <gr...@acm.org>.
Subversion's features are really driven by its customers. Right now,
there are two groups customers: Collab.net's customers and the open source
customers of Subversion (GCC seems to be a big one).
If commercial users really want 'svn obliterate', they would pay for it.
If open source users really want 'svn obliterate', it would come about
probably
as the result of a lawsuit or something.
Based on both the bugzilla entry and emails in this group, it sounds
like 'svn obliterate' is non-trivial both from an implementation and
use-case perspective.
Quoting Nik Clayton <ni...@ngo.org.uk>:
> Hi,
>
> Is anyone actively working on implementing an API to completely
> remove a file from a repository, as described in:
>
> http://subversion.tigris.org/issues/show_bug.cgi?id=516
>
> ?
>
> I'm looking at what would be involved in migrating FreeBSD from CVS
> to Subversion, and we think we need this functionality.
>
> As well as the traditional "Someone's committed a core file by
> accident" use case that's covered in that issue, we have another use
> case that doesn't seem to have been discussed (at least, a quick
> search didn't turn it up).
>
> This is where a committer inadvertently commits code that they are
> not licensed to commit. This might be a binary blob from a third
> party vendor that's not supposed to be distributed, or it might be,
> as has happened to FreeBSD, been code being developed for a
> commercial third party, that will eventually be open sourced, but
> that is not supposed to be open sourced yet.
>
> That's happened to FreeBSD. Some emergency CVS repo surgery meant
> that the offending code was removed from the master repo, and all the
> mirrors shortly caught up.
>
> I don't (yet) have a solution that will work with Subversion. I'm
> aware that it's possible to dump the repo from just before and just
> after the commit, load the first dump, manufacture a dummy commit,
> and then load the second dump.
>
> However, I don't yet know how feasible that is. First, the dump/load
> cycle for a large repository is lengthy -- doing this to a
> hypothetical FreeBSD SVN repo, which will contain 10+ years of
> imported CVS history is likely to be prohibitively so. Second, I
> haven't tested what the effects of this procedure would be on any one
> who has mirrored the repo (using svnsync) or has a working copy that
> contains the offending item.
>
> Alternative approaches, such as restore from a backup prior to the
> commit, and replay the remaining commits are also possible, but
> again, I've yet to investigate what sort of effect this would have on
> mirrors that are svnsync(1)'ing the repo.
>
> Hence our interest in supported command/API that does this.
>
> N
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: dev-help@subversion.tigris.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Issues 516: "svn obliterate"
Posted by Nik Clayton <ni...@ngo.org.uk>.
Folker Schamel wrote:
> Nik Clayton wrote:
[...]
>> A few hours later (after being woken up at unfriendly time of the
>> morning), one of the master repository managers directly edited the
>> underlying ,v files -- the new files were completely removed, the
>> modified files were reset so that not only did the commit never happen,
>> there was no trace of the commit ever happening.
>
> You want "svn obliterate" to not work on per-file basis,
> but on per-commit basis?
Yes. If a particular commit contains a mix of files that we are allowed
to distribute and files that we're not allowed to distribute it is
simpler to obliterate the commit, and then recommit the correct files.
This also handles the case where bad code is committed to a file that
previously existed. I wouldn't want to remove that file from the entire
repo history, I just want to remove that change.
> What should "svn obliterate" do if a later regular commit
> affects the same file as the inadvertent commit?
Consider a repo that contains foo.c at r1. Someone inadvertently
commits proprietry code to foo.c at r2, and then someone else comes
along and commits more code to foo.c in r3.
I'd like r2 to disappear. By this I mean that the delta of the change
between r1 and r2, and between r2 and r3, in this repo (and any repos
synced from this repo) must not contain the proprietry code[1].
If the changes in r2 and r3 did not affect the same lines in the file
(i.e., there's no conflict) then this is probably quite simple. Just
flag r2 as being 'empty', and make sure the delta is 'empty' (padded
with NULs, or similar).
This might leave you with a 'bad' file, in the sense that the code might
no longer compile -- perhaps r2 included a new function, and r3 contains
code to call the function. But this is no different from a botched
merge, and can be worked around in subsequent commits.
If the changes in r2 and r3 are dependent (say r2 adds a function, and
r3 modifies that function) then "svn obliterate" should detect this, and
warn the user with a list of all the dependent revisions, and then exit.
% svn obliterate -r 2 $REPO
Revision 3 depends on revision 2
Cannot remove revision 2 due to dependent revisions
There are probably two ways to handle what then has to happen.
Either "svn obliterate --force" could then be used. The effect would be
to obliterate the requested revision, and any revisions that depend on it.
% svn obliterate --force -r 2 $REPO
Revision 3 depends on revision 2
Obliterated revision 3
Obliterated revision 2
Or, the user runs "svn obliterate", and passes it the complete list of
revisions identified. The command removes the revisions in order,
youngest first
% svn obliterate -r 2 -r 3 $REPO
Obliterated revision 3
Obliterated revision 2
Swapping the order of the -r options should have no effect.
These examples use 'svn', but I would be happy if this was only
available through 'svnadmin'.
N
[1] I.e., not only should it be impossible for tools built on top of the
client API to see the change, it should be impossible to create a tool
that uses the SVN filesystem API to extract the change either. If the
change still lives in the repo somehow then we're still technically
distributing code that we have no license to distribute.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Issues 516: "svn obliterate"
Posted by Grant Rettke <gr...@acm.org>.
Based on this thread, it seems like commits of unauthorized
material is a very real issue, but the issue is different for
different folks. Perhaps there ought to be a single thread that
document folks use cases for what they need in 'svn obliterate'.
Quoting Folker Schamel <sc...@spinor.com>:
> Nik Clayton wrote:
>> John Peacock wrote:
>>> Nik Clayton wrote:
>>>> However, I don't yet know how feasible that is. First, the dump/load
>>>> cycle for a large repository is lengthy -- doing this to a hypothetical
>>>> FreeBSD SVN repo, which will contain 10+ years of imported CVS history
>>>> is likely to be prohibitively so.
>>>
>>> You should also realize that if you use cvs2svn to convert your CVS
>>> repository,
>>> that generates a Subversion dumpfile as part of its processing. You
>>> can use
>>> svndumpfilter on that file to eliminate all of the paths that should
>>> go missing
>>> before you load it into a Subversion repo...
>>
>> Right -- but that's for the one time import. I'm more concerned about
>> events that happen after that.
>>
>> To paraphrase from a real-world FreeBSD example (names changed).
>>
>> We had a committer who was employed by a US company that uses FreeBSD.
>> That company was developing some updates to FreeBSD code. The company
>> wanted to use that code in its next product release, and, some time
>> later, donate this proprietry code back to FreeBSD under the BSD license.
>>
>> This committer inadvertently committed this code back to the FreeBSD
>> tree. This code consisted of some new files, and some changes to
>> existing source code.
>>
>> Within minutes this code, which we were not licensed to distribute, was
>> mirrored to other CVS servers around the world (there are 162 of them at
>> the moment).
>>
>> A few hours later (after being woken up at unfriendly time of the
>> morning), one of the master repository managers directly edited the
>> underlying ,v files -- the new files were completely removed, the
>> modified files were reset so that not only did the commit never happen,
>> there was no trace of the commit ever happening.
>
> You want "svn obliterate" to not work on per-file basis,
> but on per-commit basis?
>
> What should "svn obliterate" do if a later regular commit
> affects the same file as the inadvertent commit?
>
> Cheers,
> Folker
>
>>
>> Because of the way CVS works, these changes could then be propogated to
>> the other mirrors, and integrated directly in to the ,v files, with no
>> changes to (or notification required to) the mirrors.
>>
>> Big US company is happy, and stops threatening us with lawyers.
>>
>> I need a way to solve that problem, quickly, should it ever happen again
>> if FreeBSD moves to Subversion.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: dev-help@subversion.tigris.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Issues 516: "svn obliterate"
Posted by Folker Schamel <sc...@spinor.com>.
Nik Clayton wrote:
> John Peacock wrote:
>> Nik Clayton wrote:
>>> However, I don't yet know how feasible that is. First, the dump/load
>>> cycle for a large repository is lengthy -- doing this to a hypothetical
>>> FreeBSD SVN repo, which will contain 10+ years of imported CVS history
>>> is likely to be prohibitively so.
>>
>> You should also realize that if you use cvs2svn to convert your CVS
>> repository,
>> that generates a Subversion dumpfile as part of its processing. You
>> can use
>> svndumpfilter on that file to eliminate all of the paths that should
>> go missing
>> before you load it into a Subversion repo...
>
> Right -- but that's for the one time import. I'm more concerned about
> events that happen after that.
>
> To paraphrase from a real-world FreeBSD example (names changed).
>
> We had a committer who was employed by a US company that uses FreeBSD.
> That company was developing some updates to FreeBSD code. The company
> wanted to use that code in its next product release, and, some time
> later, donate this proprietry code back to FreeBSD under the BSD license.
>
> This committer inadvertently committed this code back to the FreeBSD
> tree. This code consisted of some new files, and some changes to
> existing source code.
>
> Within minutes this code, which we were not licensed to distribute, was
> mirrored to other CVS servers around the world (there are 162 of them at
> the moment).
>
> A few hours later (after being woken up at unfriendly time of the
> morning), one of the master repository managers directly edited the
> underlying ,v files -- the new files were completely removed, the
> modified files were reset so that not only did the commit never happen,
> there was no trace of the commit ever happening.
You want "svn obliterate" to not work on per-file basis,
but on per-commit basis?
What should "svn obliterate" do if a later regular commit
affects the same file as the inadvertent commit?
Cheers,
Folker
>
> Because of the way CVS works, these changes could then be propogated to
> the other mirrors, and integrated directly in to the ,v files, with no
> changes to (or notification required to) the mirrors.
>
> Big US company is happy, and stops threatening us with lawyers.
>
> I need a way to solve that problem, quickly, should it ever happen again
> if FreeBSD moves to Subversion.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Issues 516: "svn obliterate"
Posted by Nik Clayton <ni...@ngo.org.uk>.
John Peacock wrote:
> Nik Clayton wrote:
>> However, I don't yet know how feasible that is. First, the dump/load
>> cycle for a large repository is lengthy -- doing this to a hypothetical
>> FreeBSD SVN repo, which will contain 10+ years of imported CVS history
>> is likely to be prohibitively so.
>
> You should also realize that if you use cvs2svn to convert your CVS repository,
> that generates a Subversion dumpfile as part of its processing. You can use
> svndumpfilter on that file to eliminate all of the paths that should go missing
> before you load it into a Subversion repo...
Right -- but that's for the one time import. I'm more concerned about
events that happen after that.
To paraphrase from a real-world FreeBSD example (names changed).
We had a committer who was employed by a US company that uses FreeBSD.
That company was developing some updates to FreeBSD code. The company
wanted to use that code in its next product release, and, some time
later, donate this proprietry code back to FreeBSD under the BSD license.
This committer inadvertently committed this code back to the FreeBSD
tree. This code consisted of some new files, and some changes to
existing source code.
Within minutes this code, which we were not licensed to distribute, was
mirrored to other CVS servers around the world (there are 162 of them at
the moment).
A few hours later (after being woken up at unfriendly time of the
morning), one of the master repository managers directly edited the
underlying ,v files -- the new files were completely removed, the
modified files were reset so that not only did the commit never happen,
there was no trace of the commit ever happening.
Because of the way CVS works, these changes could then be propogated to
the other mirrors, and integrated directly in to the ,v files, with no
changes to (or notification required to) the mirrors.
Big US company is happy, and stops threatening us with lawyers.
I need a way to solve that problem, quickly, should it ever happen again
if FreeBSD moves to Subversion.
N
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Issues 516: "svn obliterate"
Posted by John Peacock <jp...@rowman.com>.
Nik Clayton wrote:
> However, I don't yet know how feasible that is. First, the dump/load
> cycle for a large repository is lengthy -- doing this to a hypothetical
> FreeBSD SVN repo, which will contain 10+ years of imported CVS history
> is likely to be prohibitively so.
You should also realize that if you use cvs2svn to convert your CVS repository,
that generates a Subversion dumpfile as part of its processing. You can use
svndumpfilter on that file to eliminate all of the paths that should go missing
before you load it into a Subversion repo...
John
--
John Peacock
Director of Information Research and Technology
Rowman & Littlefield Publishing Group
4501 Forbes Blvd
Suite H
Lanham, MD 20706
301-459-3366 x.5010
fax 301-429-5747
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Issues 516: "svn obliterate"
Posted by Nik Clayton <ni...@ngo.org.uk>.
Marcus Rueckert wrote:
> svnadmin dump repos | svndumpfilter exclude offendingpath | svnadmin load newrepos
>
> http://subversion.tigris.org/faq.html#removal
Did you read my message? I explicitly suggested that, and then
suggested that on a repo with 10+ years of history that is likely to be
a slow process. Since the repository is effectively locked while this
happens, all your committers are effectively prevented from committing
until this process completes.
Also, the effect of this on any mirrors of the repository (using
svnsync) is currently unknown. If this requires all the mirrors (and
the FreeBSD project has hundreds) to throw away their repository and
resync from scratch then that is probably a non-starter.
N
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Issues 516: "svn obliterate"
Posted by Daniel Rall <dl...@collab.net>.
Nik, AFAIK, no one is actively working on this feature ATM. A design
proposal that could be committed to notes/ would be a great start.
Re: Issues 516: "svn obliterate"
Posted by Marcus Rueckert <da...@web.de>.
hi,
svnadmin dump repos | svndumpfilter exclude offendingpath | svnadmin load newrepos
http://subversion.tigris.org/faq.html#removal
arix
--
openSUSE - SUSE Linux is my linux
openSUSE is good for you
www.opensuse.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org