Re: Unshelving through the new WC editor API

[Julian Foad <ju...@apache.org>]

Julian Foad wrote on 2019-01-14:
> I'm starting to re-implement "unshelve" using the new WC editor API. [...]
> What I'll want to do now is drive a WC-mods-editor with the changes found: [...]

That's committed and working.

Next I'll want to use the editor API for pushing changes *into* a shelf. And there's not much point in that unless I also upgrade the shelf storage to support all or more of the possible ("committable") changes that the editor API supports -- especially operations on directories (mkdir, rmdir) and copies.

I put my thoughts on IRC the other day, like this:

I was starting to rewrite the shelf storage to be driven by a delta editor, and at the same time thinking how to make it able to store "copy" operations. That's becoming more and more like a real WC. Realizing how complex it is to write that from scratch, I'm now thinking it's time to ditch the current storage and use a "real" WC to store a shelf, as the next development iteration.

Now, at last, a "real" WC supports delta-editor for input (svn_client__wc_editor) and output (svn_client__wc_replay) of local-modifications. That means it's becoming feasible to "simply" copy changes from one to another -- the "svn x-wc-copy-mods" command implements this and (mostly) works. I don't claim it's complete and bug-free yet. I think "svn_client__wc_replay" mostly is complete and correct, but svn_client__wc_editor isn't. But it's fairly close.

And this isn't everything we need to implement decent shelving. The next thing we need is ability to copy a WC-base. Copying a WC-base can be done partly with the (unfinished) "svn info --x-viewspec" thing, which basically serializes a "reporter" drive. I have an idea to look into, about how to implement efficiently the receiver end of that.

And, to make shelving better, we want to augment the "local mods editor". Currently, based on delta_editor_t, it only supports committable changes. That means shelving won't preserve "local move" info, as well as all other uncommittable WC states (conflicts, changelists, unversioned files, etc.) To support uncommittable WC state, we need to upgrade to another API that extends or augments svn_delta_editor_t adding those other things. Ultimately, I believe every kind of state that a WC can store should be able to be input and output (symmetrically) through such an API.

Back to the local-mods delta-editor support. One more piece is still needed. To implement the delta-editor "copy" operation, we need a way to fetch the base of the copy. The delta-editor API doesn't provide this; it assumes its user will be able to use any copy-from-URL to contact a repository and reference the copy-base in there. I propose that the first implementation of shelving using delta-editor will do just that: contact the repository whenever it needs the base of a copy.

Later I can formalize an extension to the delta-editor API, that's something like a callback for fetching a subtree from a source that isn't necessarily a repository but could be a WC or shelf instead, in a standard way.

There's a more major issue with wc_editor at the moment. I haven't been clear whether it's meant to apply edits against the WC base state, or whether it's meant to apply edits against the current working state. The latter is what I've mainly meant it to do, but I need to check it's consistent. And I haven't been clear to what extent it should attempt to do merging, if the existing WC state differs from the base state of the given edits. Maybe more than one variant will be needed.

Baby steps, in order:
  (1) Shelve committable changes, with very simple capture of WC base state (maybe assumes single-revision, for example): a good enough first step.
  (2) Better capture of base state.
  (3) Better replay of "copy" operations, reading WC local storage to fetch their base rather than contacting the repo.
  (4) Support some uncommittable changes such as "local move".

Etc. Each step extends the relevant interface, which is a single interface definition and two or more implementations of it (into-wc, out-of-wc, and maybe into-shelf, out-of-shelf if a shelf is not equal to a WC).

Right now I'm starting to implement Shelving v3 using a WC for shelf storage. To copy changes between shelf storage WC and user's WC, it uses the equivalent of "svn x-wc-copy-mods" which is implemented with svn_client__wc_replay() piped to svn_client__wc_editor().

Using a whole WC for each shelf-version is space-inefficient. I'm putting up with that, in the initial implementation, for the sake of developing the right architecture and APIs, which is my priority. Because this isn't just about creating some kind of shelving that works well enough to be usable (though that's nice to have).

-- 
- Julian

Re: Shelving v3 started

[Julian Foad <ju...@apache.org>]

> >   (2) Better capture of base state.

[Summarizing from IRC.]

Today I'm trying to figure out how to get Shelving v3 from last week's initial demo state (email "Shelving v3 started", branch "shelving-v3") to something that could be considered experimentally usable in a v1.12 release. My overall approach is to aim for high level APIs to do the required functional blocks; I'm not going to hack the wc-db.

The current state is to make a shelf we:
  - copy the whole (user's) WC to somewhere else where it will become the shelf storage for this shelf-version;
  - revert all the modifications in that copy so it is now a copy of the base state of the user's WC;
  - to shelve the *requested* changes we use the new high-level wc-copy-mods API to copy the requested local modifications from the user's WC to this shelf storage WC;
  - Finally, if requested ("shelve" versus "shelf-save"), we revert the requested mods from the user's WC.

The bit that needs work next is making a copy of the base state of the user's WC.

Here's the plan for that bit:

(1) Create the WC base checkout by using existing "checkout" code, initially driven from the repository, using URLs and revnums read from the user's WC (as in "svn info --x-viewspec").
(1a) For the whole WC; (1b) trimmed down to a given path specification.

(2) Create the WC base reading API (limited to a given path-spec) and think of a way to test/exercise it in isolation.

(3) Wire up (1) and (2) together.


For (1): The existing high level API to create a WC with a given base state (and no local modifications) is the client side of "checkout". Of course we only need to copy the base of locally modified nodes in the requested portions of the user's WC, not the whole WC. So we need an API that creates (a bit like checkout) a new WC that includes just those paths. I wonder if we could do that by using the "checkout" client-side machinery, unmodified, to create the WC, being driven from a "pretend Repository Access API" driver that feeds "real" data for the given paths and sets all other paths to depth-exclude.

For (2) the base-reading API: The existing high level API to read the base state of a WC is ... seems to be missing. "export" is but a small subset of it, lacking properties and repository URLs and revisions metadata. I can investigate starting from existing "diff -r0:BASE" code and see if that's useful. However, it writes directly to the diff-tree-processor API and I'm not sure if that's convertible to editor API.

Moving the shelf-storage-WCs location back inside .svn/experimental/... will be trivial after step (1a).

-- 
- Julian

Shelving v3 started

[Julian Foad <ju...@apache.org>]

On the 'shelving-v3' branch:

* shelving uses a separate 'real' WC to store each shelf

* copying local modifications between the (main) WC and a shelf is, at last, done the Right Way using an Editor API:
    - "open(user's WC).replay_local_mods()" piped to "open(shelf-storage-wc).local_mods_editor()".

Benefits:

* every kind of committable change will be supported

* no shelf-specific code for storing and manipulating changes, just high level glue code

Problems:

* bug in reverting shelved changes from the main WC after shelving them:
    - added nodes don't get deleted from disk, even with 'svn revert --remove-added'
    - https://issues.apache.org/jira/browse/SVN-4798
    - causing XFAIL in shelf_tests 18 "shelve mkdir" and 20 "shelve replace dir"

* unshelve doesn't attempt to merge
    - assumes it's writing to a WC that matches the shelf base state
    - causing XFAIL in shelf_tests for merge and conflict tests (25, 26, 27, 28, 30)

* copying the WC base state is crude, space-inefficient
    - currently using a simple recursive copy of the entire WC dir including '.svn' metadata
    - currently puts shelves outside the main WC, in a sibling dir named, "<wc-dir>.shelves";
      that's just to avoid infinite recursion when doing the recursive dir copy, and could be
      changed to "<wc-dir>/.svn/experimental/shelves/v3" with a little tweak.

The only reason this is committed to a branch rather than trunk is that it is writing to a directory outside the WC. It would be worth tweaking that back inside ".svn" as soon as possible, in order to move the development back onto trunk.


Julian Foad wrote on 2019-02-01:
[...]
> Baby steps, in order:
>   (1) Shelve committable changes, with very simple capture of WC base 
> state (maybe assumes single-revision, for example): a good enough first 
> step.

Yay, working!

>   (2) Better capture of base state.
>   (3) Better replay of "copy" operations, reading WC local storage to 
> fetch their base rather than contacting the repo.
>   (4) Support some uncommittable changes such as "local move".
[...]


-- 
- Julian