You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Greg Stein <gs...@gmail.com> on 2012/02/01 05:08:17 UTC

Ev2, RA, Commit Process (was: Editor v2 - suggestions and queries)

I've included an email from back in November for the basis of the
"next steps" for Ev2 tweaks. As I mentioned yesterday, Hyrum and I got
a chance to talk at length about Ev2 issues and design. Some extra
conversations with Philip and Julian also helped out here.

These next steps feel Good, but have some big implications around
client/server interaction. I'd like to hear any thoughts and concerns.

See below:

On Sat, Nov 5, 2011 at 19:56, Greg Stein <gs...@gmail.com> wrote:
> On Fri, Nov 4, 2011 at 11:16, Julian Foad <ju...@wandisco.com> wrote:
>...
>> Huh?  We're now talking about a single call that sets the target and
>> the properties together.  If we take this approach, I suggest naming
>> the three calls 'set_symlink' (like 'add_symlink'), 'set_file' (like
>> 'add_file'), and 'set_dir_props' (which is not quite like
>> 'add_directory').
>
> It was never intended to be set_dir_props(). You could set properties
> on any node. That should stick around, cuz it kind of sucks for the
> caller to have to know the node type just to set some properties.
>
> But I do like where you're going with this. Rather than set_$kind,
> let's go with alter_file(), alter_symlink(), and alter_directory(). I
> chose "alter" rather than "change" since change_directory might throw
> people off with some implied stateful semantics. Each of the alter_*
> functions would provide for changing the properties, symlink target,
> file contents, etc. Maybe we eliminate alter_directory() since that
> would be exactly the same as set_props() [though maybe that becomes
> alter_props?].

I plan to eliminate set_props() and its COMPLETE parameter. Instead,
there will be three APIs for changing a node, and they will complete
before returning. Thus, no more dual calls where a receiver may need
to retain state to accomplish the entire change. The receiver can
perform the change to the node, then throw out any temporary state.
The node won't be touched again.

These entry points will be:

alter_directory(RELPATH, REVISION, PROPS)
alter_file(RELPATH, REVISION, PROPS, CHECKSUM, CONTENTS)
alter_symlink(RELPATH, REVISION, PROPS, TARGET)

Again, we use "alter" to avoid change_directory to avoid confusion
with the standard "chdir()" posix call.

set_props, set_text, and set_target all go away.

One more detail about add_file() and alter_file() is noted below.

>...
>> I wanted to clarify three separate things here.
>>
>>  (1) Partial read is allowed.  Good.
>>
>>  (2) It's a 'pull-mode' interface.  Fine.
>>
>>  (3) The editor is not allowed to return early and defer the reading
>> of this stream until it's ready.  I wonder if we might want to let the
>> editor keep several streams open and read from them as and when its
>> transmit buffer allows, especially if it wants to be able to send two
>> or more file streams in parallel.  These are just shallow thoughts at
>> the moment.
>
> Actually... damn. One of the reasons for a separate of set_props() vs
> set_text() was to allow for the delayed delivery of contents. Same
> thing for the add_file() and set_text(). We adjusted add_file() to
> take contents, but that may have been a mistake.
>
> At commit time, we want to delay the delivery of the content streams.

Actually, this will be fine. Hyrum and I figured out a different
(better?) approach. This allows the contents to always be provided at
add_file() and alter_file() time, rather than allowing a delayed
delivery. As noted above, two-step interfaces and delayed calls can
make it difficult for a receiver -- they need to retain some state
about the node and link up the two calls to complete the
addition/change.

>...
> Well... see above, ref: delayed content delivery. The API as I
> originally designed provided for a delayed delivery. I forgot about
> that aspect when I acceded to combining add_file/set_text.

Again, we will leave it as-is, and alter_file() will also contain the contents.

When Hyrum and I went through this, there are two occasions when file
contents are delayed:

1) at commit time, we note all the changes that will be made, expect a
"fast-fail" from the server for out-of-date items, and then we start
delivery of the bulk/file content

2) at update time, the changes to a file may be noted in the update
report skeleton, applied via the editor, and then a separate GET is
run to fetch the contents and then set the contents when it arrives
(via a delayed apply_textdelta). (Note: for Neon, with its
mother-report approach, the content is typically present at the time
of the file's metadata changes)

For problem #2, we will simply make the RA update process (as an Ev2
driver) manage the delayed state, rather than impose the burden upon
all Ev2 receivers.

For problem #1, it gets trickier. Hyrum noted that the delayed content
delivery exists *only* so that we can get the fast-fail on the commit
process, and then suggested: why don't we simply tell the server the
entire commit plan, get the response, and *then* start sending all the
changes to the server?

Thus: we propose to turn the commit process into two parts, and
corresponding RA interfaces:

Step 1: The commit process tells RA something like "here are all the
relpaths/revisions that I plan to $operation". Note that it isn't
really "all paths" since recursive operations like a copy-destination
path, or a deletion, don't need to list all the child nodes. This
"plan" is sent to the server, which starts a txn, and examines whether
any of the operations are being applied to out-of-date nodes. The
server can allow the commit operation to proceed, or respond with an
error (possibly, multiple errors!).

Step 2: The commit process then drives an Ev2 commit editor to send
all the changes to the server. This is a blend of metadata changes and
content delivery (no delayed content, as before).

The "plan" is some new XML report-like document, posted to the "me"
resource on the server to create the FS txn and perform the check. I'm
not sure what the schema looks like, what kinds of data items are
needed, nor what the RA API looks like. This "plan" is probably an
opaque object constructed by the commit process. It would be nice to
have this in libsvn_ra, and the internals available to all RA layers.
This plan object may be able to replace the "commit info" stuff that
we have in the client today (preferable).

For backwards compatibility, the RA API still needs to provide for the
old commit process. That *may* be mappable to the new server "plan"
protocol, but I'm not sure. The RA layer may need to retain too much
information in memory (specifically: properties), until the
apply_textdelta calls arrive with the content (the first one signaling
the end of the Step 1, and the beginning of Step 2). It can probably
use the Ev1 interface calls to construct a plan, and then use the new
"send-plan" machinery at the transition stage. This compatibility code
may be able to live in libsvn_ra and be implemented in terms of the
new RA APIs (plan + Ev2). Thus, all RA layers may be able to get rid
of their Ev1 code and just implement plan+Ev2.

A new client talking to an old server would simply make Step 1 (the
plan) perform all the working resource checkouts, which is how an old
server performs the out-of-date checks. When Step 2 is run, the
changes to the working resources are performed.

.... okay. That's my brain dump for now.

Thoughts?

Cheers,
-g

Re: Ev2, RA, Commit Process (was: Editor v2 - suggestions and queries)

Posted by Branko Čibej <br...@apache.org>.
On 01.02.2012 21:22, C. Michael Pilato wrote:
> Yup. libsvn_delta is the appropriate place for the commit plan stuff.

Completely forgot about that part, sorry.

> Though, I'd prefer to give it a more generic naming if possible, as I
> don't see anything explicitly commit-specific about it. An "edit
> prelude" or "preview" or "abstract" or somesuch. 

Well ... we have batons, so this could be a "score" and the Ev2 the
"conductor" ... then we can ask people who report bugs, "what's the
score?" :)

-- Brane

Re: Ev2, RA, Commit Process (was: Editor v2 - suggestions and queries)

Posted by "C. Michael Pilato" <cm...@collab.net>.
On 02/01/2012 03:18 PM, Greg Stein wrote:
>> If the delta editors are currently client-side, and we suddenly need
>> server-side code, too ... maybe a new library is in order?
> 
> The basic code for svn_delta_editor_t is in libsvn_delta. eg.
> default_editor, cancellation editor, etc. Plus svndiff and xdelta.
> Some other bits, probably.
> 
> We use the delta editor in libsvn_repos (the dir_deltas stuff). The
> editor drive is then marshalled into (eg.) the update report response.
> IOW, libsvn_delta is already used on both client and server.

Yup.  libsvn_delta is the appropriate place for the commit plan stuff.
Though, I'd prefer to give it a more generic naming if possible, as I don't
see anything explicitly commit-specific about it.  An "edit prelude" or
"preview" or "abstract" or somesuch.

-- 
C. Michael Pilato <cm...@collab.net>
CollabNet   <>   www.collab.net   <>   Distributed Development On Demand


Re: Ev2, RA, Commit Process (was: Editor v2 - suggestions and queries)

Posted by Greg Stein <gs...@gmail.com>.
On Wed, Feb 1, 2012 at 15:10, Branko Čibej <br...@apache.org> wrote:
> On 01.02.2012 20:34, Greg Stein wrote:
>> On Wed, Feb 1, 2012 at 07:57, Julian Foad <ju...@btopenworld.com> wrote:
>>> Greg Stein wrote:
>>>
>>>> Oh. And that is an interesting point. We need server-side processing
>>>> of a commit plan. Maybe the plan code goes down into libsvn_subr so
>>>> both sides can create/consume/process the plans. [...]
>>> That's what libsvn_delta's for, no?
>> Oh. Yeah, I guess it could go in there, where libsvn_delta is "diff
>> code, and describing and making changes to trees". Something like
>> that. And a commit plan falls into that.
>
> I thought libsvn_delta was strictly contextless binary deltas ... xdelta
> reimplemented, more or less. We even have a separate libsvn_diff.
>
> Clearly libsvn_subr is a catch-all, but tree walking code hardly fits in
> there.
>
> If the delta editors are currently client-side, and we suddenly need
> server-side code, too ... maybe a new library is in order?

The basic code for svn_delta_editor_t is in libsvn_delta. eg.
default_editor, cancellation editor, etc. Plus svndiff and xdelta.
Some other bits, probably.

We use the delta editor in libsvn_repos (the dir_deltas stuff). The
editor drive is then marshalled into (eg.) the update report response.
IOW, libsvn_delta is already used on both client and server.

(forgot about libsvn_diff)

Cheers,
-g

Re: Ev2, RA, Commit Process (was: Editor v2 - suggestions and queries)

Posted by Branko Čibej <br...@apache.org>.
On 01.02.2012 20:34, Greg Stein wrote:
> On Wed, Feb 1, 2012 at 07:57, Julian Foad <ju...@btopenworld.com> wrote:
>> Greg Stein wrote:
>>
>>> Oh. And that is an interesting point. We need server-side processing
>>> of a commit plan. Maybe the plan code goes down into libsvn_subr so
>>> both sides can create/consume/process the plans. [...]
>> That's what libsvn_delta's for, no?
> Oh. Yeah, I guess it could go in there, where libsvn_delta is "diff
> code, and describing and making changes to trees". Something like
> that. And a commit plan falls into that.

I thought libsvn_delta was strictly contextless binary deltas ... xdelta
reimplemented, more or less. We even have a separate libsvn_diff.

Clearly libsvn_subr is a catch-all, but tree walking code hardly fits in
there.

If the delta editors are currently client-side, and we suddenly need
server-side code, too ... maybe a new library is in order?

-- Brane

Re: Ev2, RA, Commit Process (was: Editor v2 - suggestions and queries)

Posted by Greg Stein <gs...@gmail.com>.
On Wed, Feb 1, 2012 at 07:57, Julian Foad <ju...@btopenworld.com> wrote:
> Greg Stein wrote:
>
>> Oh. And that is an interesting point. We need server-side processing
>> of a commit plan. Maybe the plan code goes down into libsvn_subr so
>> both sides can create/consume/process the plans. [...]
>
> That's what libsvn_delta's for, no?

Oh. Yeah, I guess it could go in there, where libsvn_delta is "diff
code, and describing and making changes to trees". Something like
that. And a commit plan falls into that.

Cheers,
-g

Re: Ev2, RA, Commit Process (was: Editor v2 - suggestions and queries)

Posted by Julian Foad <ju...@btopenworld.com>.
Greg Stein wrote:

> Oh. And that is an interesting point. We need server-side processing
> of a commit plan. Maybe the plan code goes down into libsvn_subr so
> both sides can create/consume/process the plans. [...]

That's what libsvn_delta's for, no?

- Julian

Re: Ev2, RA, Commit Process (was: Editor v2 - suggestions and queries)

Posted by Daniel Shahaf <da...@elego.de>.
Greg Stein wrote on Wed, Feb 01, 2012 at 03:07:56 -0500:
> On Wed, Feb 1, 2012 at 02:28, Daniel Shahaf <da...@elego.de> wrote:
> > Also, you didn't talk about ra_svn at all.  Presumably it's trivial
> > there (we can just extend the protocol to mirror the new APIs), but
> > explicitness won't hurt.
> 
> I presume it should be pretty easy, but have no idea how the svn
> protocol does capability negotiation or gets extended.
> 

There is support for server capabilities and client capabilities.
Extension options are:

    4. Extensibility
    ----------------

    This protocol may be extended in three ways, in decreasing order of
    desirability:

      * Items may be added to any tuple.  An old implementation will
        ignore the extra items.

      * Named extensions may be expressed at connection initiation time
        by the client or server.

      * The protocol version may be bumped.  Clients and servers can then
        choose to any range of protocol versions.

> IIRC, the delta editor interface simply gets marshaled over the wire.
> Maybe we just pass a plan-skel over the wire? Oh. Wait. svn protocol
> doesn't actually use skels, does it? Something similar, but entirely
> different if I remember right. (SIGH) ... I guess we can always embed

Yes.  It doesn't use skels but another similar syntax with a slightly
different marshalling syntax.

> a serialized skel within svn's other-funky serialization format.
> 
> Oh. And that is an interesting point. We need server-side processing
> of a commit plan. Maybe the plan code goes down into libsvn_subr so
> both sides can create/consume/process the plans. I suspect that
> server-side processing would go into libsvn_repos and use existing FS
> APIs to pass/fail a commit plan.
> 
> Cheers,
> -g

Re: Ev2, RA, Commit Process (was: Editor v2 - suggestions and queries)

Posted by Greg Stein <gs...@gmail.com>.
On Wed, Feb 1, 2012 at 02:28, Daniel Shahaf <da...@elego.de> wrote:
> Greg Stein wrote on Tue, Jan 31, 2012 at 23:08:17 -0500:
>> The "plan" is some new XML report-like document, posted to the "me"
>> resource
>
> We've defined HTTPv2 such that the body of POSTs is skels, not XML.
> (dav_svn__method_post())

Ah. Right. I was thinking of the report processing. The skel approach
is much nicer. Thanks for the reminder!

> Also, you didn't talk about ra_svn at all.  Presumably it's trivial
> there (we can just extend the protocol to mirror the new APIs), but
> explicitness won't hurt.

I presume it should be pretty easy, but have no idea how the svn
protocol does capability negotiation or gets extended.

IIRC, the delta editor interface simply gets marshaled over the wire.
Maybe we just pass a plan-skel over the wire? Oh. Wait. svn protocol
doesn't actually use skels, does it? Something similar, but entirely
different if I remember right. (SIGH) ... I guess we can always embed
a serialized skel within svn's other-funky serialization format.

Oh. And that is an interesting point. We need server-side processing
of a commit plan. Maybe the plan code goes down into libsvn_subr so
both sides can create/consume/process the plans. I suspect that
server-side processing would go into libsvn_repos and use existing FS
APIs to pass/fail a commit plan.

Cheers,
-g

Re: Ev2, RA, Commit Process (was: Editor v2 - suggestions and queries)

Posted by Daniel Shahaf <da...@elego.de>.
Greg Stein wrote on Tue, Jan 31, 2012 at 23:08:17 -0500:
> The "plan" is some new XML report-like document, posted to the "me"
> resource

We've defined HTTPv2 such that the body of POSTs is skels, not XML.
(dav_svn__method_post())

Also, you didn't talk about ra_svn at all.  Presumably it's trivial
there (we can just extend the protocol to mirror the new APIs), but
explicitness won't hurt.

> Thoughts?
> 
> Cheers,
> -g

Re: Ev2, RA, Commit Process (was: Editor v2 - suggestions and queries)

Posted by Greg Stein <gs...@gmail.com>.
On Wed, Feb 1, 2012 at 00:03, Hyrum K Wright <hy...@wandisco.com> wrote:
>...
> It does require some state to be maintained, but not as much as it
> appears at first blush.
>
> In the current delta editor, we have to keep all these file batons
> around, since we need to tie a particular apply_txdelta call to a
> given file.  If we deal in the currency of hashes, all that state goes
> away.  The add_file/alter_file callbacks have told us the content they
> expect, and we trust that at some future date that content will get to
> the destination.
>
> Because the content has essentially a UUID, we don't need to maintain
> per-file state while we wait for it to arrive, we just need a simple
> list of things we're expecting, and a way to ensure they get sent.
> "Here's the tree; here's the content; I'll let you sort out what
> matches to what."  Or even "Here's the tree"  "Thanks, I need the
> following content"  "Okay, here's the content you want."
>
> Does that make sense?

Yup. But it still seems a bit more complicated.

I guess this approach means that we could keep a delayed content
delivery and not mess around with the commit process, but I do like
using the Plan to obviate delivery of certain content streams. Was
that your thinking here? (minimal impact to commit) Or is there
something a bit more? (and does that benefit outweight the extra
checksum/content matching step)

Thanks,
-g

Re: Ev2, RA, Commit Process (was: Editor v2 - suggestions and queries)

Posted by Hyrum K Wright <hy...@wandisco.com>.
On Tue, Jan 31, 2012 at 10:55 PM, Greg Stein <gs...@gmail.com> wrote:
> On Tue, Jan 31, 2012 at 23:27, Hyrum K Wright <hy...@wandisco.com> wrote:
>>...
>> One of the purposes the "commit plan" could accomplish is establishing
>> a priori what content the server has, and what content we need to
>> send, identified via sha1.  Thus, if a commit is something like a
>> merge, where most of the content is already on the server, the client
>> doesn't have to transmit that information again.  (Though maybe this
>> is already served by the lazy-load semantics of the Ev2 contents
>> streams.)
>
> Ah. Right. Forgot about that part.
>
> Yes, the server can say "don't send C1, C2, C3 to me". Typically, that
> list will be empty, but sure: sometimes we can save delivery of
> content. Also note that the list of checksums [to not send] needs to
> be preserved from Step 1 to Step 2 of the commit process. Some sort of
> stateful commit process object needs to be carried between the two RA
> steps.
>
>> Another (complimentary?) approach might be to totally divorce content
>> from structure.  Just remove contents streams from Ev2 completely.
>
> I'm not sure that I like this approach. That goes back to the "save
> state across two API invocations" that we had before. It also means
> every Ev2 receiver will need to have some out-of-band way to receive
> the content or manage an interaction with a "checksum to content"
> mapping subsystem.
>
> Sounds yucky, or I'm not understanding your suggestion.

It does require some state to be maintained, but not as much as it
appears at first blush.

In the current delta editor, we have to keep all these file batons
around, since we need to tie a particular apply_txdelta call to a
given file.  If we deal in the currency of hashes, all that state goes
away.  The add_file/alter_file callbacks have told us the content they
expect, and we trust that at some future date that content will get to
the destination.

Because the content has essentially a UUID, we don't need to maintain
per-file state while we wait for it to arrive, we just need a simple
list of things we're expecting, and a way to ensure they get sent.
"Here's the tree; here's the content; I'll let you sort out what
matches to what."  Or even "Here's the tree"  "Thanks, I need the
following content"  "Okay, here's the content you want."

Does that make sense?

-Hyrum



-- 

uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com/

Re: Ev2, RA, Commit Process (was: Editor v2 - suggestions and queries)

Posted by Greg Stein <gs...@gmail.com>.
On Tue, Jan 31, 2012 at 23:27, Hyrum K Wright <hy...@wandisco.com> wrote:
>...
> One of the purposes the "commit plan" could accomplish is establishing
> a priori what content the server has, and what content we need to
> send, identified via sha1.  Thus, if a commit is something like a
> merge, where most of the content is already on the server, the client
> doesn't have to transmit that information again.  (Though maybe this
> is already served by the lazy-load semantics of the Ev2 contents
> streams.)

Ah. Right. Forgot about that part.

Yes, the server can say "don't send C1, C2, C3 to me". Typically, that
list will be empty, but sure: sometimes we can save delivery of
content. Also note that the list of checksums [to not send] needs to
be preserved from Step 1 to Step 2 of the commit process. Some sort of
stateful commit process object needs to be carried between the two RA
steps.

> Another (complimentary?) approach might be to totally divorce content
> from structure.  Just remove contents streams from Ev2 completely.

I'm not sure that I like this approach. That goes back to the "save
state across two API invocations" that we had before. It also means
every Ev2 receiver will need to have some out-of-band way to receive
the content or manage an interaction with a "checksum to content"
mapping subsystem.

Sounds yucky, or I'm not understanding your suggestion.

>...

Cheers,
-g

Re: Ev2, RA, Commit Process (was: Editor v2 - suggestions and queries)

Posted by Hyrum K Wright <hy...@wandisco.com>.
On Tue, Jan 31, 2012 at 10:08 PM, Greg Stein <gs...@gmail.com> wrote:
> I've included an email from back in November for the basis of the
> "next steps" for Ev2 tweaks. As I mentioned yesterday, Hyrum and I got
> a chance to talk at length about Ev2 issues and design. Some extra
> conversations with Philip and Julian also helped out here.
>
> These next steps feel Good, but have some big implications around
> client/server interaction. I'd like to hear any thoughts and concerns.
>
> See below:
>
> On Sat, Nov 5, 2011 at 19:56, Greg Stein <gs...@gmail.com> wrote:
>> On Fri, Nov 4, 2011 at 11:16, Julian Foad <ju...@wandisco.com> wrote:
>>...
>>> Huh?  We're now talking about a single call that sets the target and
>>> the properties together.  If we take this approach, I suggest naming
>>> the three calls 'set_symlink' (like 'add_symlink'), 'set_file' (like
>>> 'add_file'), and 'set_dir_props' (which is not quite like
>>> 'add_directory').
>>
>> It was never intended to be set_dir_props(). You could set properties
>> on any node. That should stick around, cuz it kind of sucks for the
>> caller to have to know the node type just to set some properties.
>>
>> But I do like where you're going with this. Rather than set_$kind,
>> let's go with alter_file(), alter_symlink(), and alter_directory(). I
>> chose "alter" rather than "change" since change_directory might throw
>> people off with some implied stateful semantics. Each of the alter_*
>> functions would provide for changing the properties, symlink target,
>> file contents, etc. Maybe we eliminate alter_directory() since that
>> would be exactly the same as set_props() [though maybe that becomes
>> alter_props?].
>
> I plan to eliminate set_props() and its COMPLETE parameter. Instead,
> there will be three APIs for changing a node, and they will complete
> before returning. Thus, no more dual calls where a receiver may need
> to retain state to accomplish the entire change. The receiver can
> perform the change to the node, then throw out any temporary state.
> The node won't be touched again.
>
> These entry points will be:
>
> alter_directory(RELPATH, REVISION, PROPS)
> alter_file(RELPATH, REVISION, PROPS, CHECKSUM, CONTENTS)
> alter_symlink(RELPATH, REVISION, PROPS, TARGET)
>
> Again, we use "alter" to avoid change_directory to avoid confusion
> with the standard "chdir()" posix call.
>
> set_props, set_text, and set_target all go away.
>
> One more detail about add_file() and alter_file() is noted below.
>
>>...
>>> I wanted to clarify three separate things here.
>>>
>>>  (1) Partial read is allowed.  Good.
>>>
>>>  (2) It's a 'pull-mode' interface.  Fine.
>>>
>>>  (3) The editor is not allowed to return early and defer the reading
>>> of this stream until it's ready.  I wonder if we might want to let the
>>> editor keep several streams open and read from them as and when its
>>> transmit buffer allows, especially if it wants to be able to send two
>>> or more file streams in parallel.  These are just shallow thoughts at
>>> the moment.
>>
>> Actually... damn. One of the reasons for a separate of set_props() vs
>> set_text() was to allow for the delayed delivery of contents. Same
>> thing for the add_file() and set_text(). We adjusted add_file() to
>> take contents, but that may have been a mistake.
>>
>> At commit time, we want to delay the delivery of the content streams.
>
> Actually, this will be fine. Hyrum and I figured out a different
> (better?) approach. This allows the contents to always be provided at
> add_file() and alter_file() time, rather than allowing a delayed
> delivery. As noted above, two-step interfaces and delayed calls can
> make it difficult for a receiver -- they need to retain some state
> about the node and link up the two calls to complete the
> addition/change.
>
>>...
>> Well... see above, ref: delayed content delivery. The API as I
>> originally designed provided for a delayed delivery. I forgot about
>> that aspect when I acceded to combining add_file/set_text.
>
> Again, we will leave it as-is, and alter_file() will also contain the contents.
>
> When Hyrum and I went through this, there are two occasions when file
> contents are delayed:
>
> 1) at commit time, we note all the changes that will be made, expect a
> "fast-fail" from the server for out-of-date items, and then we start
> delivery of the bulk/file content
>
> 2) at update time, the changes to a file may be noted in the update
> report skeleton, applied via the editor, and then a separate GET is
> run to fetch the contents and then set the contents when it arrives
> (via a delayed apply_textdelta). (Note: for Neon, with its
> mother-report approach, the content is typically present at the time
> of the file's metadata changes)
>
> For problem #2, we will simply make the RA update process (as an Ev2
> driver) manage the delayed state, rather than impose the burden upon
> all Ev2 receivers.
>
> For problem #1, it gets trickier. Hyrum noted that the delayed content
> delivery exists *only* so that we can get the fast-fail on the commit
> process, and then suggested: why don't we simply tell the server the
> entire commit plan, get the response, and *then* start sending all the
> changes to the server?
>
> Thus: we propose to turn the commit process into two parts, and
> corresponding RA interfaces:
>
> Step 1: The commit process tells RA something like "here are all the
> relpaths/revisions that I plan to $operation". Note that it isn't
> really "all paths" since recursive operations like a copy-destination
> path, or a deletion, don't need to list all the child nodes. This
> "plan" is sent to the server, which starts a txn, and examines whether
> any of the operations are being applied to out-of-date nodes. The
> server can allow the commit operation to proceed, or respond with an
> error (possibly, multiple errors!).
>
> Step 2: The commit process then drives an Ev2 commit editor to send
> all the changes to the server. This is a blend of metadata changes and
> content delivery (no delayed content, as before).
>
> The "plan" is some new XML report-like document, posted to the "me"
> resource on the server to create the FS txn and perform the check. I'm
> not sure what the schema looks like, what kinds of data items are
> needed, nor what the RA API looks like. This "plan" is probably an
> opaque object constructed by the commit process. It would be nice to
> have this in libsvn_ra, and the internals available to all RA layers.
> This plan object may be able to replace the "commit info" stuff that
> we have in the client today (preferable).
>
> For backwards compatibility, the RA API still needs to provide for the
> old commit process. That *may* be mappable to the new server "plan"
> protocol, but I'm not sure. The RA layer may need to retain too much
> information in memory (specifically: properties), until the
> apply_textdelta calls arrive with the content (the first one signaling
> the end of the Step 1, and the beginning of Step 2). It can probably
> use the Ev1 interface calls to construct a plan, and then use the new
> "send-plan" machinery at the transition stage. This compatibility code
> may be able to live in libsvn_ra and be implemented in terms of the
> new RA APIs (plan + Ev2). Thus, all RA layers may be able to get rid
> of their Ev1 code and just implement plan+Ev2.
>
> A new client talking to an old server would simply make Step 1 (the
> plan) perform all the working resource checkouts, which is how an old
> server performs the out-of-date checks. When Step 2 is run, the
> changes to the working resources are performed.
>
> .... okay. That's my brain dump for now.
>
> Thoughts?

I've had a few since last week, so I'll throw 'em out there.

One of the purposes the "commit plan" could accomplish is establishing
a priori what content the server has, and what content we need to
send, identified via sha1.  Thus, if a commit is something like a
merge, where most of the content is already on the server, the client
doesn't have to transmit that information again.  (Though maybe this
is already served by the lazy-load semantics of the Ev2 contents
streams.)

Another (complimentary?) approach might be to totally divorce content
from structure.  Just remove contents streams from Ev2 completely.
We'd still have the checksums, and so Ev2 would still be making edits
to trees in a well-defined way, we'd just be leaving the actual
transmittal of the checksum to some other process, which may or
may-not be in band.  In this paradigm, Ev2 goes back to being the
"commit plan", and we still get post-fix text deltas.  Though, given
our current filesystem limitations, this design might not be doable.

(All this, of course, makes my backward compat shims even more
heinous, but as long as we're redesigning The World, let's look at it
from as many angles as reasonable.)

-Hyrum


-- 

uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com/