Obliterate - milestone 1 reached: the most basic obliteration

[Julian Foad <ju...@wandisco.com>]

Hi, fans of "obliterate".

I am pleased to announce that a very restricted form of "svn obliterate"
is now working.

According to the plan
<http://svn.apache.org/repos/asf/subversion/trunk/notes/obliterate/plan-milestones.html>, I intended to get a simple kind of obliteration (last version of a node) working by the end of January. Something like this is working now. Let me explain exactly what the status is.

The present "svn obliterate" command in trunk (which is compiled in if
the macro SVN_WITH_EXPERIMENTAL_OBLITERATE is defined) can now remove a
file node from the head revision. Yes, only from the head revision,
which isn't much use in a busy repository, I know.

This is implemented only in BDB as yet; I am deferring FSFS support till
later because development for BDB is more efficient in terms of my
effort. I will update the plan to reflect this change.

The three attached text files show the changes that "svn obliterate"
makes to the BDB database, and are best viewed in a side-by-side diff
program such as vimdiff or kdiff3.

'bdb-before.txt' is the repository with 10 revisions, before
obliterating anything. 'bdb-after.txt' is that repository after
obliterating the files. 'bdb-simulated.txt' is a repository created in
the same was as the 'before' repository, except deleting the files just
before committing revision 10, thus ending up with the same repository
content that the obliteration is supposed to produce, only the various
ids in the database are different. Each file is a dump of the BDB
tables, as written out by obliterate_tests.py, and then hand-edited,
mainly to remove some redundant rows that are still in the database but
don't contribute to the active content of the repository.

Please take a look at the differences between these three files if you
will. I think they show two things.

(1) They show that the 'copies' information in the 'transactions' table
is not properly updated: the transaction still thinks it has a node that
is a copy (node-rev 'i.1.f' copied from '/f-mov/E@9'), but in fact that
node-rev was obliterated and so the copy information should be deleted
too.

(2) Even after hand editing, they show lots of redundant new table rows,
especially reps and strings:

  * unreferenced ones

  * rows identical to earlier existing ones

  * directory reps that have a string to represent an empty directory
instead of just no string.

(1) is a bug; (2) is room for optimization but not critical.


NEXT STEPS

More important than (1) and (2) is extending the support to obliterating
older revisions. My first step there is to make it adjust any immediate
successor (the same node id in the next revision) to take account of the
change. After that I will look at searching for all future copies of the
node, which is facilitated by the "successor-ids" table as implemented
in the "fs-successor-ids" branch.

Philip Martin has looked at an important angle of development which is
to obliterate a file's content rather than the existence of the node
itself. Deleting a file's content by replacing its content
representation in the DB can be done far less invasively (with far less
complexity of implementation) than removing a directory entry, and at
the same time this changes all successive revisions of the file and all
copies of it, which is often what the user wants. That provides an
efficient way of obliterating the same file across a large range of
revisions. However there are some difficulties with it. I will be
investigating how best to gain the advantages of that approach.


- Julian

Re: Obliterate - milestone 1 reached: the most basic obliteration

[Julian Foad <ju...@wandisco.com>]

On Thu, 2010-02-04 at 17:25 +0100, Lieven Govaerts wrote:
> On Thu, Jan 28, 2010 at 1:00 PM, Julian Foad <ju...@wandisco.com> wrote:
> > Hi, fans of "obliterate".
> >
> > I am pleased to announce that a very restricted form of "svn obliterate"
> > is now working.
> >
> > According to the plan
> > <http://svn.apache.org/repos/asf/subversion/trunk/notes/obliterate/plan-milestones.html>, I intended to get a simple kind of obliteration (last version of a node) working by the end of January. Something like this is working now. Let me explain exactly what the status is.
> >
> > The present "svn obliterate" command in trunk (which is compiled in if
> > the macro SVN_WITH_EXPERIMENTAL_OBLITERATE is defined) can now remove a
> > file node from the head revision. Yes, only from the head revision,
> > which isn't much use in a busy repository, I know.
> >
> > This is implemented only in BDB as yet; I am deferring FSFS support till
> > later because development for BDB is more efficient in terms of my
> > effort. I will update the plan to reflect this change.
> >
> [..]
> >
> > NEXT STEPS
> >
> > More important than (1) and (2) is extending the support to obliterating
> > older revisions. My first step there is to make it adjust any immediate
> > successor (the same node id in the next revision) to take account of the
> > change. After that I will look at searching for all future copies of the
> > node, which is facilitated by the "successor-ids" table as implemented
> > in the "fs-successor-ids" branch.
> >
> > Philip Martin has looked at an important angle of development which is
> > to obliterate a file's content rather than the existence of the node
> > itself. Deleting a file's content by replacing its content
> > representation in the DB can be done far less invasively (with far less
> > complexity of implementation) than removing a directory entry, and at
> > the same time this changes all successive revisions of the file and all
> > copies of it, which is often what the user wants. That provides an
> > efficient way of obliterating the same file across a large range of
> > revisions. However there are some difficulties with it. I will be
> > investigating how best to gain the advantages of that approach.
> >
> 
> Just wanted to say: keep up the good work.
> 
> I'll try to follow your changes a bit more, as soon as you have
> something implemented in fsfs.
> 
> In the meantime, if I find a bit of time I want to look at the test
> framework for the obliterate tests, as I'm a bit worried it will be
> difficult to test all the obliterate details efficiently with our
> current framework.

Thanks, and I would certainly benefit from any assistance with testing
you can give.

- Julian

Re: Obliterate - milestone 1 reached: the most basic obliteration

[Lieven Govaerts <sv...@mobsol.be>]

On Thu, Jan 28, 2010 at 1:00 PM, Julian Foad <ju...@wandisco.com> wrote:
> Hi, fans of "obliterate".
>
> I am pleased to announce that a very restricted form of "svn obliterate"
> is now working.
>
> According to the plan
> <http://svn.apache.org/repos/asf/subversion/trunk/notes/obliterate/plan-milestones.html>, I intended to get a simple kind of obliteration (last version of a node) working by the end of January. Something like this is working now. Let me explain exactly what the status is.
>
> The present "svn obliterate" command in trunk (which is compiled in if
> the macro SVN_WITH_EXPERIMENTAL_OBLITERATE is defined) can now remove a
> file node from the head revision. Yes, only from the head revision,
> which isn't much use in a busy repository, I know.
>
> This is implemented only in BDB as yet; I am deferring FSFS support till
> later because development for BDB is more efficient in terms of my
> effort. I will update the plan to reflect this change.
>
[..]
>
> NEXT STEPS
>
> More important than (1) and (2) is extending the support to obliterating
> older revisions. My first step there is to make it adjust any immediate
> successor (the same node id in the next revision) to take account of the
> change. After that I will look at searching for all future copies of the
> node, which is facilitated by the "successor-ids" table as implemented
> in the "fs-successor-ids" branch.
>
> Philip Martin has looked at an important angle of development which is
> to obliterate a file's content rather than the existence of the node
> itself. Deleting a file's content by replacing its content
> representation in the DB can be done far less invasively (with far less
> complexity of implementation) than removing a directory entry, and at
> the same time this changes all successive revisions of the file and all
> copies of it, which is often what the user wants. That provides an
> efficient way of obliterating the same file across a large range of
> revisions. However there are some difficulties with it. I will be
> investigating how best to gain the advantages of that approach.
>

Just wanted to say: keep up the good work.

I'll try to follow your changes a bit more, as soon as you have
something implemented in fsfs.

In the meantime, if I find a bit of time I want to look at the test
framework for the obliterate tests, as I'm a bit worried it will be
difficult to test all the obliterate details efficiently with our
current framework.

Lieven

Re: Obliterate - milestone 1 reached: the most basic obliteration

[Julian Foad <ju...@btopenworld.com>]

On Thu, 2010-01-28, Julian Foad wrote:
> Please take a look at the differences between these three files if you
> will. I think they show two things.
> 
> (1) They show that the 'copies' information in the 'transactions' table
> is not properly updated: the transaction still thinks it has a node that
> is a copy (node-rev 'i.1.f' copied from '/f-mov/E@9'), but in fact that
> node-rev was obliterated and so the copy information should be deleted
> too.

I fixed (1) in r905283.

- Julian