You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Mark Phippard <ma...@gmail.com> on 2012/09/26 15:13:44 UTC
General question on Serf
I have a question about how Serf works that I keep forgetting to send to
this list.
When you do an update with Serf, my understanding is that it does a REPORT
request that gives it back a list of files that it needs to update and then
it fires off a bunch of GET requests to retrieve those files. With Neon,
the changes are all included in the REPORT response.
My question is how this intersects with Deltas.
Suppose I have a large text file in my working copy and someone has made a
small minor change to the file. When using Neon, I know that the client
and server just exchange a relatively small delta that is applied to my
working copy. I imagine this delta is part of the REPORT response that the
client receives.
What happens with Serf? From what I know, I would assume that Serf has to
do a GET of the entire file. Is this true or does it somehow only request
a delta? And what might this mean for real world performance.
Kind of a side question. One of the benefits of the Serf design is that
theoretically enables the usage of caching proxies between the client and
server. This is another reason I assume that Serf must request the entire
file. I do not see how a caching proxy would provide much benefit if the
Serf requests were for somewhat arbitrary deltas between two revisions.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
RE: General question on Serf
Posted by Bert Huijben <be...@qqmail.nl>.
I think I can answer this question, but maybe one of the ra_serf gurus can
fill in the details.
With the get request serf sends a header to tell the server which version of
the file the client already knows. The server than usually sends the delta,
just like neon or svnserve do/did. The server is allowed to send the whole
file, but our implementation doesn't do this.
(We used to send the whole files early in HTTPv2 development because we
somehow missed this header and things got a lot faster when we fixed this).
Bert
From: Mark Phippard [mailto:markphip@gmail.com]
Sent: woensdag 26 september 2012 15:14
To: Subversion Development
Subject: General question on Serf
I have a question about how Serf works that I keep forgetting to send to
this list.
When you do an update with Serf, my understanding is that it does a REPORT
request that gives it back a list of files that it needs to update and then
it fires off a bunch of GET requests to retrieve those files. With Neon,
the changes are all included in the REPORT response.
My question is how this intersects with Deltas.
Suppose I have a large text file in my working copy and someone has made a
small minor change to the file. When using Neon, I know that the client and
server just exchange a relatively small delta that is applied to my working
copy. I imagine this delta is part of the REPORT response that the client
receives.
What happens with Serf? From what I know, I would assume that Serf has to
do a GET of the entire file. Is this true or does it somehow only request a
delta? And what might this mean for real world performance.
Kind of a side question. One of the benefits of the Serf design is that
theoretically enables the usage of caching proxies between the client and
server. This is another reason I assume that Serf must request the entire
file. I do not see how a caching proxy would provide much benefit if the
Serf requests were for somewhat arbitrary deltas between two revisions.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
Re: General question on Serf
Posted by Greg Stein <gs...@gmail.com>.
On Wed, Sep 26, 2012 at 2:11 PM, Philip Martin
<ph...@wandisco.com> wrote:
> Lieven Govaerts <sv...@mobsol.be> writes:
>
>> If the intention is to instruct the proxy that a cached response is
>> only valid for a certain X-SVN-VR-Base header, then that is what the
>> "Vary" header is for.
>> So with:
>> Vary: X-SVN-VR-Base
>> in the response, the proxy should only return a cached response if the
>> request for that cached response had the same value for X-SVN-VR-Base
>> as the new request.
>
> Sounds like we really ought to do that. I've raised
> http://subversion.tigris.org/issues/show_bug.cgi?id=4235
Yup. That's the right answer, and it appears that mod_dav_svn does
*not* do that. I've scanned through httpd, mod_dav, and mod_dav_svn
... there aren't any good helpers for managing the Vary: header :-(
But yes: that header is what proxies (should) use for determining
whether a response is based varies on header values, in addition to
the URI itself.
Cheers,
-g
Re: General question on Serf
Posted by Philip Martin <ph...@wandisco.com>.
Lieven Govaerts <sv...@mobsol.be> writes:
> If the intention is to instruct the proxy that a cached response is
> only valid for a certain X-SVN-VR-Base header, then that is what the
> "Vary" header is for.
> So with:
> Vary: X-SVN-VR-Base
> in the response, the proxy should only return a cached response if the
> request for that cached response had the same value for X-SVN-VR-Base
> as the new request.
Sounds like we really ought to do that. I've raised
http://subversion.tigris.org/issues/show_bug.cgi?id=4235
--
Certified & Supported Apache Subversion Downloads:
http://www.wandisco.com/subversion/download
Re: General question on Serf
Posted by Lieven Govaerts <sv...@mobsol.be>.
On Wed, Sep 26, 2012 at 7:27 PM, C. Michael Pilato <cm...@collab.net> wrote:
> On 09/26/2012 12:28 PM, Philip Martin wrote:
>> Branko Čibej <br...@wandisco.com> writes:
>>
>>> This is how the cache /should/ work, according to any number of HTTP
>>> specs. Unfortunately, most proxies that I know about don't attempt
>>> anything as "advanced" as that.
>>
>> If the cache ignores X-SVN-VR-Base and simply caches the response that
>> will break 1.8 clients. Looking at the response:
>>
>> HTTP/1.1 200 OK
>> Date: Wed, 26 Sep 2012 16:20:33 GMT
>> Server: Apache/2.2.16 (Debian) mod_ssl/2.2.16 OpenSSL/0.9.8o DAV/2 SVN/1.8.0-dev
>> Last-Modified: Wed, 26 Sep 2012 16:20:19 GMT
>> ETag: "2//f"
>> Cache-Control: max-age=604800
>> Accept-Ranges: bytes
>> Transfer-Encoding: chunked
>> Content-Type: application/vnd.svn-svndiff
>>
>> I don't see a header that allows the client to confirm that that delta
>> is based on the requested revision. I suppose the client just assumes
>> that the server used the right base revision so if a cache returns the
>> wrong delta the client will get a checksum mismatch when constructing
>> the full-text.
>
> That's right. The client only cares about that Content-Type header -- if
> "application/vnd.svn-svndiff", the content is expected to be a delta against
> the revision in the request's X-SVN-VR-Base header.
>
>> Perhaps the server should send the X-SVN-VR-Base header back?
>
> Wouldn't be a bad idea. At a minimum, a client could compare it with the
> one that was submitted in the GET request, and error out if they don't match.
>
If the intention is to instruct the proxy that a cached response is
only valid for a certain X-SVN-VR-Base header, then that is what the
"Vary" header is for.
So with:
Vary: X-SVN-VR-Base
in the response, the proxy should only return a cached response if the
request for that cached response had the same value for X-SVN-VR-Base
as the new request.
Lieven
Re: General question on Serf
Posted by "C. Michael Pilato" <cm...@collab.net>.
On 09/26/2012 12:28 PM, Philip Martin wrote:
> Branko Čibej <br...@wandisco.com> writes:
>
>> This is how the cache /should/ work, according to any number of HTTP
>> specs. Unfortunately, most proxies that I know about don't attempt
>> anything as "advanced" as that.
>
> If the cache ignores X-SVN-VR-Base and simply caches the response that
> will break 1.8 clients. Looking at the response:
>
> HTTP/1.1 200 OK
> Date: Wed, 26 Sep 2012 16:20:33 GMT
> Server: Apache/2.2.16 (Debian) mod_ssl/2.2.16 OpenSSL/0.9.8o DAV/2 SVN/1.8.0-dev
> Last-Modified: Wed, 26 Sep 2012 16:20:19 GMT
> ETag: "2//f"
> Cache-Control: max-age=604800
> Accept-Ranges: bytes
> Transfer-Encoding: chunked
> Content-Type: application/vnd.svn-svndiff
>
> I don't see a header that allows the client to confirm that that delta
> is based on the requested revision. I suppose the client just assumes
> that the server used the right base revision so if a cache returns the
> wrong delta the client will get a checksum mismatch when constructing
> the full-text.
That's right. The client only cares about that Content-Type header -- if
"application/vnd.svn-svndiff", the content is expected to be a delta against
the revision in the request's X-SVN-VR-Base header.
> Perhaps the server should send the X-SVN-VR-Base header back?
Wouldn't be a bad idea. At a minimum, a client could compare it with the
one that was submitted in the GET request, and error out if they don't match.
--
C. Michael Pilato <cm...@collab.net>
CollabNet <> www.collab.net <> Enterprise Cloud Development
Re: General question on Serf
Posted by Philip Martin <ph...@wandisco.com>.
Branko Čibej <br...@wandisco.com> writes:
> This is how the cache /should/ work, according to any number of HTTP
> specs. Unfortunately, most proxies that I know about don't attempt
> anything as "advanced" as that.
If the cache ignores X-SVN-VR-Base and simply caches the response that
will break 1.8 clients. Looking at the response:
HTTP/1.1 200 OK
Date: Wed, 26 Sep 2012 16:20:33 GMT
Server: Apache/2.2.16 (Debian) mod_ssl/2.2.16 OpenSSL/0.9.8o DAV/2 SVN/1.8.0-dev
Last-Modified: Wed, 26 Sep 2012 16:20:19 GMT
ETag: "2//f"
Cache-Control: max-age=604800
Accept-Ranges: bytes
Transfer-Encoding: chunked
Content-Type: application/vnd.svn-svndiff
I don't see a header that allows the client to confirm that that delta
is based on the requested revision. I suppose the client just assumes
that the server used the right base revision so if a cache returns the
wrong delta the client will get a checksum mismatch when constructing
the full-text.
Perhaps the server should send the X-SVN-VR-Base header back?
I suppose the server admin might be able to use mod_headers to send
Cache-Control: no-cache
if a dodgy cache is known to be affecting clients, although a dodgy
cache might ignore that.
--
Certified & Supported Apache Subversion Downloads:
http://www.wandisco.com/subversion/download
Re: General question on Serf
Posted by Branko Čibej <br...@wandisco.com>.
On 26.09.2012 17:48, Philip Martin wrote:
> Mark Phippard <ma...@gmail.com> writes:
>
>> I have always been skeptical of the ability to put a cache in front of the
>> SVN server. Wouldn't something like this keep a cache from working
>> properly? How would a cache know about that header we added and that the
>> content we returned to the client cannot be cached? I presume we set
>> something in the response that an intelligent cache would look at and know
>> that it cannot cache the response. But that effectively means the cache
>> would only provide some value on checkout?
> I'm no expert but I expect that the cache has to take account of the
> significant headers: it could cache multiple versions of the GET one for
> each X-SVN-VR-Base. If that is how it is expected to work then perhaps
> getting the client to use the lasst changed revision instead of the
> current revision would make the cache more effective.
This is how the cache /should/ work, according to any number of HTTP
specs. Unfortunately, most proxies that I know about don't attempt
anything as "advanced" as that.
-- Brane
--
Certified & Supported Apache Subversion Downloads:
http://www.wandisco.com/subversion/download
Re: General question on Serf
Posted by Philip Martin <ph...@wandisco.com>.
Mark Phippard <ma...@gmail.com> writes:
> I have always been skeptical of the ability to put a cache in front of the
> SVN server. Wouldn't something like this keep a cache from working
> properly? How would a cache know about that header we added and that the
> content we returned to the client cannot be cached? I presume we set
> something in the response that an intelligent cache would look at and know
> that it cannot cache the response. But that effectively means the cache
> would only provide some value on checkout?
I'm no expert but I expect that the cache has to take account of the
significant headers: it could cache multiple versions of the GET one for
each X-SVN-VR-Base. If that is how it is expected to work then perhaps
getting the client to use the lasst changed revision instead of the
current revision would make the cache more effective.
--
Certified & Supported Apache Subversion Downloads:
http://www.wandisco.com/subversion/download
Re: General question on Serf
Posted by Mark Phippard <ma...@gmail.com>.
On Wed, Sep 26, 2012 at 9:44 AM, Mark Phippard <ma...@gmail.com> wrote:
> On Wed, Sep 26, 2012 at 9:40 AM, Philip Martin <philip.martin@wandisco.com
> > wrote:
>
>> Mark Phippard <ma...@gmail.com> writes:
>>
>> > What happens with Serf? From what I know, I would assume that Serf has
>> to
>> > do a GET of the entire file. Is this true or does it somehow only
>> request
>> > a delta?
>>
>> Serf sends the delta base to the server (what does TEMPORARILY mean
>> here?):
>>
>> /** This header is *TEMPORARILY* used to transmit the delta base to the
>> * server. It contains a version resource URL for what is on the client.
>> */
>> #define SVN_DAV_DELTA_BASE_HEADER "X-SVN-VR-Base"
>>
>> so a GET looks like:
>>
>> GET /obj/repo/!svn/rvr/2/f HTTP/1.1
>> Host: localhost:8888
>> User-Agent: SVN/1.8.0-dev serf/1.1.1
>> DAV: http://subversion.tigris.org/xmlns/dav/svn/depth
>> DAV: http://subversion.tigris.org/xmlns/dav/svn/mergeinfo
>> DAV: http://subversion.tigris.org/xmlns/dav/svn/log-revprops
>> X-SVN-VR-Base<http://subversion.tigris.org/xmlns/dav/svn/log-revpropsX-SVN-VR-Base>:
>> /obj/repo/!svn/rvr/1/f
>> Accept-Encoding: svndiff1;q=0.9,svndiff;q=0.8
>>
>>
> OK, thanks. I see that is documented here:
>
>
> http://svn.apache.org/repos/asf/subversion/trunk/notes/http-and-webdav/webdav-protocol
>
>
> I assumed it must send deltas somehow or we would be noticing performance
> issues.
>
>
I have always been skeptical of the ability to put a cache in front of the
SVN server. Wouldn't something like this keep a cache from working
properly? How would a cache know about that header we added and that the
content we returned to the client cannot be cached? I presume we set
something in the response that an intelligent cache would look at and know
that it cannot cache the response. But that effectively means the cache
would only provide some value on checkout?
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
Re: General question on Serf
Posted by Mark Phippard <ma...@gmail.com>.
On Wed, Sep 26, 2012 at 9:40 AM, Philip Martin
<ph...@wandisco.com>wrote:
> Mark Phippard <ma...@gmail.com> writes:
>
> > What happens with Serf? From what I know, I would assume that Serf has
> to
> > do a GET of the entire file. Is this true or does it somehow only
> request
> > a delta?
>
> Serf sends the delta base to the server (what does TEMPORARILY mean
> here?):
>
> /** This header is *TEMPORARILY* used to transmit the delta base to the
> * server. It contains a version resource URL for what is on the client.
> */
> #define SVN_DAV_DELTA_BASE_HEADER "X-SVN-VR-Base"
>
> so a GET looks like:
>
> GET /obj/repo/!svn/rvr/2/f HTTP/1.1
> Host: localhost:8888
> User-Agent: SVN/1.8.0-dev serf/1.1.1
> DAV: http://subversion.tigris.org/xmlns/dav/svn/depth
> DAV: http://subversion.tigris.org/xmlns/dav/svn/mergeinfo
> DAV: http://subversion.tigris.org/xmlns/dav/svn/log-revprops
> X-SVN-VR-Base: /obj/repo/!svn/rvr/1/f
> Accept-Encoding: svndiff1;q=0.9,svndiff;q=0.8
>
>
OK, thanks. I see that is documented here:
http://svn.apache.org/repos/asf/subversion/trunk/notes/http-and-webdav/webdav-protocol
I assumed it must send deltas somehow or we would be noticing performance
issues.
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
Re: General question on Serf
Posted by Philip Martin <ph...@wandisco.com>.
Philip Martin <ph...@wandisco.com> writes:
> GET /obj/repo/!svn/rvr/2/f HTTP/1.1
> Host: localhost:8888
> User-Agent: SVN/1.8.0-dev serf/1.1.1
> DAV: http://subversion.tigris.org/xmlns/dav/svn/depth
> DAV: http://subversion.tigris.org/xmlns/dav/svn/mergeinfo
> DAV: http://subversion.tigris.org/xmlns/dav/svn/log-revprops
> X-SVN-VR-Base: /obj/repo/!svn/rvr/1/f
> Accept-Encoding: svndiff1;q=0.9,svndiff;q=0.8
I see we are still using "tigris" in the headers. Is this something we
could/should change to "apache"?
Do we need to send those DAV: headers when the request is a GET? If I
use telnet and construct a GET manually they don't appear to be needed.
It's adding over 150 bytes onto every GET request.
--
Certified & Supported Apache Subversion Downloads:
http://www.wandisco.com/subversion/download
Re: General question on Serf
Posted by "C. Michael Pilato" <cm...@collab.net>.
On 09/26/2012 09:51 AM, C. Michael Pilato wrote:
> On 09/26/2012 09:40 AM, Philip Martin wrote:
>> Serf sends the delta base to the server (what does TEMPORARILY mean
>> here?):
>>
>> /** This header is *TEMPORARILY* used to transmit the delta base to the
>> * server. It contains a version resource URL for what is on the client.
>> */
>> #define SVN_DAV_DELTA_BASE_HEADER "X-SVN-VR-Base"
>
> Hrm... I do wonder what "TEMPORARILY" was intended to mean there. Maybe
> there was some Delta-V-ism that we were supposed to be using but didn't?
>
> Ah! Neon's use of this header explains it:
>
> {{{
> /* The HTTP delta draft uses an If-None-Match header holding an
> entity tag corresponding to the copy we have. It is much more
> natural for us to use a version URL to specify what we have.
> Thus, we want to use the If: header to specify the URL. But
> mod_dav sees all "State-token" items as lock tokens. When we
> get mod_dav updated and the backend APIs expanded, then we
> can switch to using the If: header. For now, use a custom
> header to specify the version resource to use as the base. */
> ne_add_request_header(request->ne_req,
> SVN_DAV_DELTA_BASE_HEADER, delta_base);
> }}}
>
I've taken the liberty of porting this information into the docstring for
SVN_DAV_DELTA_BASE_HEADER for increased visibility (r1390487).
--
C. Michael Pilato <cm...@collab.net>
CollabNet <> www.collab.net <> Enterprise Cloud Development
Re: General question on Serf
Posted by "C. Michael Pilato" <cm...@collab.net>.
On 09/26/2012 09:40 AM, Philip Martin wrote:
> Serf sends the delta base to the server (what does TEMPORARILY mean
> here?):
>
> /** This header is *TEMPORARILY* used to transmit the delta base to the
> * server. It contains a version resource URL for what is on the client.
> */
> #define SVN_DAV_DELTA_BASE_HEADER "X-SVN-VR-Base"
Hrm... I do wonder what "TEMPORARILY" was intended to mean there. Maybe
there was some Delta-V-ism that we were supposed to be using but didn't?
Ah! Neon's use of this header explains it:
{{{
/* The HTTP delta draft uses an If-None-Match header holding an
entity tag corresponding to the copy we have. It is much more
natural for us to use a version URL to specify what we have.
Thus, we want to use the If: header to specify the URL. But
mod_dav sees all "State-token" items as lock tokens. When we
get mod_dav updated and the backend APIs expanded, then we
can switch to using the If: header. For now, use a custom
header to specify the version resource to use as the base. */
ne_add_request_header(request->ne_req,
SVN_DAV_DELTA_BASE_HEADER, delta_base);
}}}
--
C. Michael Pilato <cm...@collab.net>
CollabNet <> www.collab.net <> Enterprise Cloud Development
Re: General question on Serf
Posted by Philip Martin <ph...@wandisco.com>.
Mark Phippard <ma...@gmail.com> writes:
> What happens with Serf? From what I know, I would assume that Serf has to
> do a GET of the entire file. Is this true or does it somehow only request
> a delta?
Serf sends the delta base to the server (what does TEMPORARILY mean
here?):
/** This header is *TEMPORARILY* used to transmit the delta base to the
* server. It contains a version resource URL for what is on the client.
*/
#define SVN_DAV_DELTA_BASE_HEADER "X-SVN-VR-Base"
so a GET looks like:
GET /obj/repo/!svn/rvr/2/f HTTP/1.1
Host: localhost:8888
User-Agent: SVN/1.8.0-dev serf/1.1.1
DAV: http://subversion.tigris.org/xmlns/dav/svn/depth
DAV: http://subversion.tigris.org/xmlns/dav/svn/mergeinfo
DAV: http://subversion.tigris.org/xmlns/dav/svn/log-revprops
X-SVN-VR-Base: /obj/repo/!svn/rvr/1/f
Accept-Encoding: svndiff1;q=0.9,svndiff;q=0.8
--
Certified & Supported Apache Subversion Downloads:
http://www.wandisco.com/subversion/download