Pristine text storage - status

[Julian Foad <ju...@wandisco.com>]

Pristine text is now stored in the "Pristine Store" accessed by
svn_wc__db_pristine_...(), if you build with
"CFLAGS=-DSVN_EXPERIMENTAL_PRISTINE".

With one more significant change (in the attached patch) it is "close"
to working.


Things keeping it in "experimental" mode:

  * It's still suffering a few test failures - see below.

  * The WC upgrade code isn't written - see below.

  * The removal of unreferenced texts from the store is incomplete.


Things that are not blocking:

  * Switch from using MD-5 checksums as the key, to using SHA-1.

    (I think that's a nice separate task that we can do afterwards, or
even in the middle of everything else.  I have made the code store both
checksums when installing a new pristine text and accept either kind of
checksum when looking up an existing pristine text.)


PROBLEMS AND TEST FAILURES
==========================

Here's what happens running the test suite (with the attached patch):

[[[
FAIL:  db-test: Unknown test failure; see tests.log. 
  - The test tries to add a BASE_NODE checksum without first
    installing the pristine text in the store, which now fails
    an assertion.

FAIL:  basic_tests.py 40: automatic conflict resolution
  - File A/D/G/rho ends up empty after 'resolve' and 'update'.

FAIL:  diff_tests.py 4: replace a file with a file
  - Diff output is blank for file A/D/G/rho; should show a mod.

FAIL:  externals_tests.py 1: test checkouts with externals
  - workqueue.c:1918: assertion failed (src_stream != NULL)
  - a file external?

FAIL:  externals_tests.py 15: should not be able to mv or rm a file external
FAIL:  externals_tests.py 16: place a file external into a directory external
FAIL:  externals_tests.py 18: binary file externals
FAIL:  externals_tests.py 23: tag directory with file externals from wc to url

FAIL:  prop_tests.py 9: props work when reverting a replacement
  - reverting a replacement
  - due to entry_modify()?

FAIL:  revert_tests.py 3: revert a replaced file with no properties
FAIL:  revert_tests.py 5: revert svn cp PATH PATH replace file with props
FAIL:  revert_tests.py 6: revert a merge replacement of file with history
FAIL:  revert_tests.py 7: revert svn cp URL PATH replace file with props
FAIL:  revert_tests.py 8: revert file after second replace
FAIL:  revert_tests.py 18: reverted replace with history restores checksum

FAIL:  special_tests.py 7: merge symlink into file
]]]


Experience with squashing the earlier failures suggests they will all be
caused by a rather smaller number of issues.  Some of the main causes
are:

  * Some problem related to file-externals doesn't work.

  * Calls to entry_modify, on a node that is replaced, lose the checksum
recorded on the base node, and thus lose track of its pristine text.


UPGRADING OLD WORKING COPIES
============================

The next step (which can be done in parallel with fixing the above
problems) is upgrading old WCs.  I have a plan for the procedure to
upgrade an old WC:

  1. For each pristine text that is present in the old location: if its
checksum is not stored in the DB row for that node, calculate and store
its MD-5 checksum in the DB row for that node.  (Or SHA-1 if we've
switched to using SHA-1 already).

  2. For each pristine text that is present in the old location: install
a copy of it in the Pristine Store (which includes calculating the other
checksum) and then delete it from the old location.

  3. Bump the WC format number to indicate the upgrade is functionally
complete.


If anyone wants to tackle any of these, that would be great.  I'll be
around this week and on holiday next week.


- Julian

Re: Pristine text storage - status

[Julian Foad <ju...@wandisco.com>]

I (Julian Foad) wrote:
> ... that does more than just clean up unreferenced files.  Without it,
> there are many more test failures.  That's why I haven't committed that
> change yet: I need to understand it first.

Talking of test failures, a few dropped off the end of my list (that is,
I abandoned the test run before finding them all).  Here are the rest:

FAIL:  tree_conflict_tests.py 2: up/sw file: del/rpl/mv onto modify
FAIL:  update_tests.py 27: conflict markers should match the file's eol style
FAIL:  update_tests.py 33: update wc containing a replaced-with-history file
FAIL:  update_tests.py 42: update --accept automatic conflict resolution
FAIL:  update_tests.py 52: update moves and modifies a file with edits
FAIL:  upgrade_tests.py 7: test upgrading a working copy created with 1.0.0

- Julian

Re: Pristine text storage - status

[Julian Foad <ju...@wandisco.com>]

On Wed, 2010-05-26 at 14:53 +0100, Julian Foad wrote:
> Pristine text is now stored in the "Pristine Store" accessed by
> svn_wc__db_pristine_...(), if you build with
> "CFLAGS=-DSVN_EXPERIMENTAL_PRISTINE".
> 
> With one more significant change (in the attached patch) it is "close"
> to working.

There's something about that patch, presumably in the main change ...

* subversion/libsvn_wc/update_editor.c
  (close_file): When we dereference the old pristine text,
    remove it from the store.

... that does more than just clean up unreferenced files.  Without it,
there are many more test failures.  That's why I haven't committed that
change yet: I need to understand it first.

- Julian

Re: Pristine text storage - status

["Hyrum K. Wright" <hy...@mail.utexas.edu>]

On Wed, May 26, 2010 at 8:53 AM, Julian Foad <ju...@wandisco.com>wrote:

> ...
> UPGRADING OLD WORKING COPIES
> ============================
>
> The next step (which can be done in parallel with fixing the above
> problems) is upgrading old WCs.  I have a plan for the procedure to
> upgrade an old WC:
>
>  1. For each pristine text that is present in the old location: if its
> checksum is not stored in the DB row for that node, calculate and store
> its MD-5 checksum in the DB row for that node.  (Or SHA-1 if we've
> switched to using SHA-1 already).
>
>  2. For each pristine text that is present in the old location: install
> a copy of it in the Pristine Store (which includes calculating the other
> checksum) and then delete it from the old location.
>
>  3. Bump the WC format number to indicate the upgrade is functionally
> complete.
>
>
> If anyone wants to tackle any of these, that would be great.  I'll be
> around this week and on holiday next week


As discussed on IRC, I'm going to take a whack at the upgrade code.

-Hyrum