You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Justin Erenkrantz <ju...@erenkrantz.com> on 2006/02/17 10:26:05 UTC
Status of ra_serf
Here's a quick progress report as to where I am with ra_serf as of r18502.
Checkouts/updates are mostly working (a checkout of svn's trunk from
svn.collab.net works fine). No svn:externals, auth, or other
'goodies' yet. 'svn log' should work; a bug may have creeped in
cleaning up memory after a 'svn up' is completed. No, I'm not even
close to being able to run regression tests yet...
We're using the bare-bones update-report and GET/PROPFIND in order to
fetch the actual content. We're utilizing less CPU than neon overall,
but we do take a little bit longer to complete as of r18502. Our
memory usage is generally competitive with ra_dav and, in most cases,
remains constant for the duration of the checkout. (If the pipeline
length gets long enough, more requests will be in the pipeline
increasing our memory usage accordingly.)
As of r18500, we now open 4 connections to the server in order to do
the checkout. Part of the reason for this is that as we parse the
REPORT response, we can start to fetch the actual content. While we
can keep it to one connection (and have done so until just now), we'll
essentially be blocked for the length of the REPORT response: that's
not acceptable to our users, so we're going to spin up three more
connections to do the GET/PROPFINDs. (Why 4? Well, that's
IE/Firefox/etc's default per-server connection limit and is generally
considered the generally accepted maximum on the Web today.)
Even with the multiple connections, I have a feeling that ra_serf is
still not using the network as efficiently as possible. My to-do for
tomorrow is to go through network traces and see what's going on and
if there's any room for more improvements. But, with the multiple
connections, we're usually within a 10-15% of neon's checkout time at
generally less CPU time.
Informal comparison numbers (not scientific, but to give you a
ballpark of where we stand):
Extracted httpd 2.2.0 tarball from a remote machine on Mac OS X:
ra_serf checkout: 9.31s user 15.68s system 32% cpu 1:16.90 total
ra_dav checkout: 12.39s user 14.10s system 40% cpu 1:05.80 total
Checkout of httpd trunk from svn.apache.org on Linux:
ra_serf checkout: 3.98s user 5.11s system 10% cpu 1:22.89 total
ra_dav checkout: 3.67s user 2.06s system 8% cpu 1:04.02 total
Checkout of svn trunk from svn.collab.net on Mac OS X:
ra_serf checkout #1: 8.64s user 11.08s system 19% cpu 1:38.93 total
ra_serf checkout #2: 11.89s user 13.73s system 36% cpu 1:10.05 total
ra_dav checkout #1: 11.84s user 9.60s system 18% cpu 1:54.99 total
ra_dav checkout #2: 11.72s user 9.79s system 17% cpu 2:03.88 total
Hmph. Okay, I wasn't expecting ra_serf to be twice as fast; but I
swear I'm not manipulating the data! Hee-hee. As you can see, I just
ran that one twice. There could also be load spikes on svn.collab.net
too - too much variations in the time for my taste... BTW, given the
broken Apache httpd on svn.collab.net, ra_serf could end up writing
parts of a checked-out file twice - this is on my to-do list for
tomorrow to fix as well (as long as libsvn_wc supports 'whoops, let me
start that file over again' we should be okay).
Daniel, Peter: I'll address your concerns on the commits you replied
to earlier in the morning. I'm dead tired right now.
Comments welcomed.
Thanks. -- justin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Status of ra_serf
Posted by Daniel Rall <dl...@collab.net>.
On Fri, 17 Feb 2006, C. Michael Pilato wrote:
...
> I had three options:
>
> (1) Always spool all REPORT responses -- this was inefficient for
> checkouts, updates, switches, etc.
>
> (2) Give the caller the option of spooling the responses -- I couldn't
> think of a good RA-method-independent, un-implementation-detail-y
> way of exposing this option thru the RA interface.
>
> (3) Exploit the knowledge that Subversion clients tend only to do
> the full-text fetching stuff in diff and merge operations, and
> only spool REPORT responses for svn_ra_do_diff* invocations.
>
> I went with option #3. #2 would have been ideal if I'd been able to
> expose that parameter in some non-hacky way, but...
...
Mike, though it's certainly still somewhat implementation-detail-y,
the communication you describe for #2 can be accomplished through
content negotiation similar to what Dan Berlin used for svndiff1. On
the HTTP side, this would probably involve a custom HTTP header
(rather than piggy-backing on an existing Accept header). On the
custom SVN protocol side, a new capability announcement (see section
2.1 of subversion/libsvn_ra_svn/protocol).
Given the problem/performance implications of this case, introducing
this type of content negotiation seems like a reasonable compromise of
ideal behavior over a completely decoupled design.
--
Daniel Rall
Re: Status of ra_serf
Posted by "C. Michael Pilato" <cm...@collab.net>.
Justin Erenkrantz wrote:
> On 2/17/06, C. Michael Pilato <cm...@collab.net> wrote:
>
>>I'm not sure why this is a problem /you/ have, though. The spooling
>>decision/implementation code is completely within libsvn_ra_dav.
>
>
> I must be seeing something else then. What I see is httpd/mod_dav_svn
> open 'report.tmp' and write to it as it scans through the fsfs
> repository. Does that ring any bells? -- justin
That's the REPORT *request*, which is always spooled to disk (and long
pre-dates my bugfix). IIRC, we spool the report to disk because Neon
has pull interface for request bodies -- either pull the request from a
file, or pull it from a memory buffer. We'd prefer to push the request
at Neon so we can fire it off streamily.
But what I was talking about was the spooling of the REPORT *response*.
--
C. Michael Pilato <cm...@collab.net>
CollabNet <> www.collab.net <> Distributed Development On Demand
Re: Status of ra_serf
Posted by "C. Michael Pilato" <cm...@collab.net>.
Garrett Rooney wrote:
> On 2/17/06, Justin Erenkrantz <ju...@erenkrantz.com> wrote:
>
>>On 2/17/06, C. Michael Pilato <cm...@collab.net> wrote:
>>
>>>I'm not sure why this is a problem /you/ have, though. The spooling
>>>decision/implementation code is completely within libsvn_ra_dav.
>>
>>I must be seeing something else then. What I see is httpd/mod_dav_svn
>>open 'report.tmp' and write to it as it scans through the fsfs
>>repository. Does that ring any bells? -- justin
>
>
> Isn't that the replacement for the code that used to build up a
> transaction in the database to represent the working copy the client
> has?
Oh, yah. You're right.
The repository reporting code uses a file called "report.tmp".
The libsvn_ra_dav report request code uses the RA tmp-file callback,
which creates a file called "tempfile.tmp".
The libsvn_ra_dav report response code uses a file call "dav-spool".
--
C. Michael Pilato <cm...@collab.net>
CollabNet <> www.collab.net <> Distributed Development On Demand
Re: Status of ra_serf
Posted by Garrett Rooney <ro...@electricjellyfish.net>.
On 2/17/06, Justin Erenkrantz <ju...@erenkrantz.com> wrote:
> On 2/17/06, C. Michael Pilato <cm...@collab.net> wrote:
> > I'm not sure why this is a problem /you/ have, though. The spooling
> > decision/implementation code is completely within libsvn_ra_dav.
>
> I must be seeing something else then. What I see is httpd/mod_dav_svn
> open 'report.tmp' and write to it as it scans through the fsfs
> repository. Does that ring any bells? -- justin
Isn't that the replacement for the code that used to build up a
transaction in the database to represent the working copy the client
has?
-garrett
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Status of ra_serf
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On 2/17/06, C. Michael Pilato <cm...@collab.net> wrote:
> I'm not sure why this is a problem /you/ have, though. The spooling
> decision/implementation code is completely within libsvn_ra_dav.
I must be seeing something else then. What I see is httpd/mod_dav_svn
open 'report.tmp' and write to it as it scans through the fsfs
repository. Does that ring any bells? -- justin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Status of ra_serf
Posted by "C. Michael Pilato" <cm...@collab.net>.
Justin Erenkrantz wrote:
> On 2/17/06, C. Michael Pilato <cm...@collab.net> wrote:
>
>>I'm not really sure how the whole pipelining thing works, but we (I)
>>made fixes to Subversion's diff/merge code which caches the whole REPORT
>>response to disk before processing it and doing the GET thang. I did
>
>
> That's part of the problem I have - this is bad for the case where you
> do not want full-texts. The client is sitting around waiting for a
> response...Can we skip writing the REPORT response to disk if we're
> not doing full-text?
I admit that my fix violated the knowledge barriers between libraries.
I had three options:
(1) Always spool all REPORT responses -- this was inefficient for
checkouts, updates, switches, etc.
(2) Give the caller the option of spooling the responses -- I couldn't
think of a good RA-method-independent, un-implementation-detail-y
way of exposing this option thru the RA interface.
(3) Exploit the knowledge that Subversion clients tend only to do
the full-text fetching stuff in diff and merge operations, and
only spool REPORT responses for svn_ra_do_diff* invocations.
I went with option #3. #2 would have been ideal if I'd been able to
expose that parameter in some non-hacky way, but...
I'm not sure why this is a problem /you/ have, though. The spooling
decision/implementation code is completely within libsvn_ra_dav.
--
C. Michael Pilato <cm...@collab.net>
CollabNet <> www.collab.net <> Distributed Development On Demand
Re: Status of ra_serf
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On 2/17/06, C. Michael Pilato <cm...@collab.net> wrote:
> I'm not really sure how the whole pipelining thing works, but we (I)
> made fixes to Subversion's diff/merge code which caches the whole REPORT
> response to disk before processing it and doing the GET thang. I did
That's part of the problem I have - this is bad for the case where you
do not want full-texts. The client is sitting around waiting for a
response...Can we skip writing the REPORT response to disk if we're
not doing full-text?
> this because if a GET sent off in the middle of REPORT processing took
> too long to get fully handled, Apache dropped the connection on the
> REPORT (thinking "inactive client").
>
> We're not in the exact same scenario with ra_serf, but I'm wondering if
> we're not in something similar. What happens in the following scenario?
>
> c->s: REPORT request
> s->c: REPORT response
> # client starts parsing the REPORT response, and firing off GETs
> c->s: GET #1
> c->s: GET #2
> ...
> [more than httpd Timeout seconds goes by]
> c->s: GET #496
> ...
> # client finishes parsing REPORT, but that first GET long ago has
> # been sitting around unhandled for too long. Did Apache drop
> # that connection?
No. httpd's KeepAlive semantics are only about when the connection is
truly idle. TCP's Keep-Alive semantics will keep the connection open
as long as the client wants it to and httpd hasn't closed it down.
So, as soon as httpd is done with the REPORT response, it will then do
a read() on the socket - the new request is there for it to handle and
it will process up to 100 more requests before it closes the socket.
At that point, the client then has to re-queue up 101-496 (ra_serf
figures out that it should only queue up 101-200 at that point; as
before, each file is acquired through *two* HTTP requests - so halve
it). The change that I asked for in httpd's conf was that
svn.collab.net only allowed *10* HTTP requests per TCP connection.
Perhaps this'll help clarify what that change was actually about.
As I said in my earlier reply to Greg, mod_dav_svn takes *forever* to
write the update-report. There is an alternative to not using a
report (do a PROPFIND on each directory and walk the tree ourselves on
that client); but that would still likely be much much slower than
still doing the custom-report for this step. For an update where we
have some files locally, the update-report is by far the most best
solution as it lets the server know what we have and only tell us the
changed files. -- justin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Status of ra_serf
Posted by "C. Michael Pilato" <cm...@collab.net>.
Greg Hudson wrote:
> I'm actually unsure why you need more than one connection total. You
> can be sending GETs to the server as you're parsing the REPORT response,
> then start handling GET responses as soon as the REPORT response is
> done. The server should be sending data full-time, with only one
> round-trip delay at the beginning to send off the first GET request.
I'm not really sure how the whole pipelining thing works, but we (I)
made fixes to Subversion's diff/merge code which caches the whole REPORT
response to disk before processing it and doing the GET thang. I did
this because if a GET sent off in the middle of REPORT processing took
too long to get fully handled, Apache dropped the connection on the
REPORT (thinking "inactive client").
We're not in the exact same scenario with ra_serf, but I'm wondering if
we're not in something similar. What happens in the following scenario?
c->s: REPORT request
s->c: REPORT response
# client starts parsing the REPORT response, and firing off GETs
c->s: GET #1
c->s: GET #2
...
[more than httpd Timeout seconds goes by]
c->s: GET #496
...
# client finishes parsing REPORT, but that first GET long ago has
# been sitting around unhandled for too long. Did Apache drop
# that connection?
--
C. Michael Pilato <cm...@collab.net>
CollabNet <> www.collab.net <> Distributed Development On Demand
Re: Status of ra_serf
Posted by Greg Stein <gs...@lyra.org>.
On Fri, Feb 17, 2006 at 07:15:13AM -0800, Justin Erenkrantz wrote:
> On 2/17/06, Greg Hudson <gh...@mit.edu> wrote:
>...
> > I'm actually unsure why you need more than one connection total. You
> > can be sending GETs to the server as you're parsing the REPORT response,
> > then start handling GET responses as soon as the REPORT response is
> > done. The server should be sending data full-time, with only one
> > round-trip delay at the beginning to send off the first GET request.
>
> The problem is that the client is sitting around idle until the server
> finishes the REPORT response. The REPORT takes a *long* time to
Basically, the idea is keep the network *and* the local disk busy.
Preferably, keep the server not-busy, but that's a bit harder :-)
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Ok to file an issue? - Bug report regarding ..\..\.. on windows
Posted by Ivan Zhakov <ch...@gmail.com>.
On 2/21/06, Folker Schamel <sc...@spinor.com> wrote:
> Hi!
>
> Objections against filing an issue?
> (Searching for ".." didn't show up an relevant existing issue.)
I confirm this problem reproduces with trunk. I consider this is bug.
>
> Cheers,
> Folker
>
> > On Windows and with svn 1.3.0
> >
> > C:\temp\svnbugreport\d1\d2\d3>svn update ..\..\..\wc
> >
> > does not work ("Skipped '..\wc'"), whereas
> >
> > C:\temp\svnbugreport\d1\d2>svn update ..\..\wc
> >
> > works ("Updated to revision 1.").
> > I would expect that both commands are equivalent.
> > Details see below.
> >
> > Cheers, Folker
> >
> > ***** Batch file to reproduce:
> >
> > setlocal
> > rmdir c:\svnbugreport /S /Q
> > mkdir c:\svnbugreport
> > cd c:\svnbugreport
> > svn --version
> > svnadmin create repos
> > svn checkout file:///svnbugreport/repos wc
> > svn mkdir file:///svnbugreport/repos/something -m "Test"
> > mkdir d1
> > mkdir d1\d2
> > mkdir d1\d2\d3
> > cd d1\d2\d3
> > svn update ..\..\..\wc
> > cd ..
> > svn update ..\..\wc
> >
> > ***** Batch file output:
> >
> > C:\temp\svnbugreport>setlocal
> >
> > C:\temp\svnbugreport>rmdir c:\svnbugreport /S /Q
> >
> > C:\temp\svnbugreport>mkdir c:\svnbugreport
> >
> > C:\temp\svnbugreport>cd c:\svnbugreport
> >
> > C:\svnbugreport>svn --version
> > svn, version 1.3.0 (r17949)
> > compiled Jan 15 2006, 23:18:48
> >
> > Copyright (C) 2000-2005 CollabNet.
> > Subversion is open source software, see http://subversion.tigris.org/
> > This product includes software developed by CollabNet
> > (http://www.Collab.Net/).
> >
> > The following repository access (RA) modules are available:
> >
> > * ra_dav : Module for accessing a repository via WebDAV (DeltaV) protocol.
> > - handles 'http' scheme
> > - handles 'https' scheme
> > * ra_svn : Module for accessing a repository using the svn network
> > protocol.
> > - handles 'svn' scheme
> > * ra_local : Module for accessing a repository on local disk.
> > - handles 'file' scheme
> >
> >
> > C:\svnbugreport>svnadmin create repos
> >
> > C:\svnbugreport>svn checkout file:///svnbugreport/repos wc
> > Checked out revision 0.
> >
> > C:\svnbugreport>svn mkdir file:///svnbugreport/repos/something -m "Test"
> >
> > Committed revision 1.
> >
> > C:\svnbugreport>mkdir d1
> >
> > C:\svnbugreport>mkdir d1\d2
> >
> > C:\svnbugreport>mkdir d1\d2\d3
> >
> > C:\svnbugreport>cd d1\d2\d3
> >
> > C:\svnbugreport\d1\d2\d3>svn update ..\..\..\wc
> > Skipped '..\wc'
> >
> > C:\svnbugreport\d1\d2\d3>cd ..
> >
> > C:\svnbugreport\d1\d2>svn update ..\..\wc
> > A ..\..\wc\something
> > Updated to revision 1.
> >
> > *****
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> > For additional commands, e-mail: dev-help@subversion.tigris.org
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: dev-help@subversion.tigris.org
>
>
--
Ivan Zhakov
Re: Ok to file an issue? - Bug report regarding ..\..\.. on windows
Posted by Folker Schamel <sc...@spinor.com>.
> On Tue, 21 Feb 2006, Folker Schamel wrote:
>
>> Hi!
>>
>> Objections against filing an issue?
>> (Searching for ".." didn't show up an relevant existing issue.)
>>
> This seems to be issue #2347. You could add a link there if you want to.
Ah I see!
I think another link is not necessary.
Cheers, Folker
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Ok to file an issue? - Bug report regarding ..\..\.. on windows
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
On Tue, 21 Feb 2006, Folker Schamel wrote:
> Hi!
>
> Objections against filing an issue?
> (Searching for ".." didn't show up an relevant existing issue.)
>
This seems to be issue #2347. You could add a link there if you want to.
Thanks,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Ok to file an issue? - Bug report regarding ..\..\.. on windows
Posted by Folker Schamel <sc...@spinor.com>.
Hi!
Objections against filing an issue?
(Searching for ".." didn't show up an relevant existing issue.)
Cheers,
Folker
> On Windows and with svn 1.3.0
>
> C:\temp\svnbugreport\d1\d2\d3>svn update ..\..\..\wc
>
> does not work ("Skipped '..\wc'"), whereas
>
> C:\temp\svnbugreport\d1\d2>svn update ..\..\wc
>
> works ("Updated to revision 1.").
> I would expect that both commands are equivalent.
> Details see below.
>
> Cheers, Folker
>
> ***** Batch file to reproduce:
>
> setlocal
> rmdir c:\svnbugreport /S /Q
> mkdir c:\svnbugreport
> cd c:\svnbugreport
> svn --version
> svnadmin create repos
> svn checkout file:///svnbugreport/repos wc
> svn mkdir file:///svnbugreport/repos/something -m "Test"
> mkdir d1
> mkdir d1\d2
> mkdir d1\d2\d3
> cd d1\d2\d3
> svn update ..\..\..\wc
> cd ..
> svn update ..\..\wc
>
> ***** Batch file output:
>
> C:\temp\svnbugreport>setlocal
>
> C:\temp\svnbugreport>rmdir c:\svnbugreport /S /Q
>
> C:\temp\svnbugreport>mkdir c:\svnbugreport
>
> C:\temp\svnbugreport>cd c:\svnbugreport
>
> C:\svnbugreport>svn --version
> svn, version 1.3.0 (r17949)
> compiled Jan 15 2006, 23:18:48
>
> Copyright (C) 2000-2005 CollabNet.
> Subversion is open source software, see http://subversion.tigris.org/
> This product includes software developed by CollabNet
> (http://www.Collab.Net/).
>
> The following repository access (RA) modules are available:
>
> * ra_dav : Module for accessing a repository via WebDAV (DeltaV) protocol.
> - handles 'http' scheme
> - handles 'https' scheme
> * ra_svn : Module for accessing a repository using the svn network
> protocol.
> - handles 'svn' scheme
> * ra_local : Module for accessing a repository on local disk.
> - handles 'file' scheme
>
>
> C:\svnbugreport>svnadmin create repos
>
> C:\svnbugreport>svn checkout file:///svnbugreport/repos wc
> Checked out revision 0.
>
> C:\svnbugreport>svn mkdir file:///svnbugreport/repos/something -m "Test"
>
> Committed revision 1.
>
> C:\svnbugreport>mkdir d1
>
> C:\svnbugreport>mkdir d1\d2
>
> C:\svnbugreport>mkdir d1\d2\d3
>
> C:\svnbugreport>cd d1\d2\d3
>
> C:\svnbugreport\d1\d2\d3>svn update ..\..\..\wc
> Skipped '..\wc'
>
> C:\svnbugreport\d1\d2\d3>cd ..
>
> C:\svnbugreport\d1\d2>svn update ..\..\wc
> A ..\..\wc\something
> Updated to revision 1.
>
> *****
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: dev-help@subversion.tigris.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Bug report regarding ..\..\.. on windows
Posted by Folker Schamel <sc...@spinor.com>.
On Windows and with svn 1.3.0
C:\temp\svnbugreport\d1\d2\d3>svn update ..\..\..\wc
does not work ("Skipped '..\wc'"), whereas
C:\temp\svnbugreport\d1\d2>svn update ..\..\wc
works ("Updated to revision 1.").
I would expect that both commands are equivalent.
Details see below.
Cheers, Folker
***** Batch file to reproduce:
setlocal
rmdir c:\svnbugreport /S /Q
mkdir c:\svnbugreport
cd c:\svnbugreport
svn --version
svnadmin create repos
svn checkout file:///svnbugreport/repos wc
svn mkdir file:///svnbugreport/repos/something -m "Test"
mkdir d1
mkdir d1\d2
mkdir d1\d2\d3
cd d1\d2\d3
svn update ..\..\..\wc
cd ..
svn update ..\..\wc
***** Batch file output:
C:\temp\svnbugreport>setlocal
C:\temp\svnbugreport>rmdir c:\svnbugreport /S /Q
C:\temp\svnbugreport>mkdir c:\svnbugreport
C:\temp\svnbugreport>cd c:\svnbugreport
C:\svnbugreport>svn --version
svn, version 1.3.0 (r17949)
compiled Jan 15 2006, 23:18:48
Copyright (C) 2000-2005 CollabNet.
Subversion is open source software, see http://subversion.tigris.org/
This product includes software developed by CollabNet (http://www.Collab.Net/).
The following repository access (RA) modules are available:
* ra_dav : Module for accessing a repository via WebDAV (DeltaV) protocol.
- handles 'http' scheme
- handles 'https' scheme
* ra_svn : Module for accessing a repository using the svn network protocol.
- handles 'svn' scheme
* ra_local : Module for accessing a repository on local disk.
- handles 'file' scheme
C:\svnbugreport>svnadmin create repos
C:\svnbugreport>svn checkout file:///svnbugreport/repos wc
Checked out revision 0.
C:\svnbugreport>svn mkdir file:///svnbugreport/repos/something -m "Test"
Committed revision 1.
C:\svnbugreport>mkdir d1
C:\svnbugreport>mkdir d1\d2
C:\svnbugreport>mkdir d1\d2\d3
C:\svnbugreport>cd d1\d2\d3
C:\svnbugreport\d1\d2\d3>svn update ..\..\..\wc
Skipped '..\wc'
C:\svnbugreport\d1\d2\d3>cd ..
C:\svnbugreport\d1\d2>svn update ..\..\wc
A ..\..\wc\something
Updated to revision 1.
*****
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Status of ra_serf
Posted by Ryan Bloom <rb...@gmail.com>.
And, with Apache, remember that the Apache developers can't tell you
if there is a memory leak or not. All of the standard modules may be
fine, without a leak, but then add another module (mod_perl, mod_php,
etc), that allow people to run their own logic within the same
process, and a leak is absolutely possible.
Ryan
On 2/19/06, Peter Samuelson <pe...@p12n.org> wrote:
>
> [Phillip Susi]
> > Right, but is there any reason for the server not to allow an
> > unlimited number of requests per connection rather than hanging up
> > after a limit?
>
> According to apache docs a long time ago, each process only answers N
> requests, then it dies, in order to mitigate against any memory leaks
> that might happen to exist. It may be that N is too small in today's
> or tomorrow's world of deeply pipelined RPC-like requests.
>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.2 (GNU/Linux)
>
> iD8DBQFD+A1pXk7sIRPQRh0RAhISAKCfz8GD9D5isKJYotBOpeXv8znsOACgjEj5
> GHNglkWlNO8w3Tg/wkuKLSQ=
> =7kB8
> -----END PGP SIGNATURE-----
>
>
>
--
Ryan Bloom
rbb@apache.org
rbb@rkbloom.net
rbloom@gmail.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Status of ra_serf
Posted by Peter Samuelson <pe...@p12n.org>.
[Phillip Susi]
> Right, but is there any reason for the server not to allow an
> unlimited number of requests per connection rather than hanging up
> after a limit?
According to apache docs a long time ago, each process only answers N
requests, then it dies, in order to mitigate against any memory leaks
that might happen to exist. It may be that N is too small in today's
or tomorrow's world of deeply pipelined RPC-like requests.
Re: Status of ra_serf
Posted by Phillip Susi <ps...@cfl.rr.com>.
Justin Erenkrantz wrote:
>
> You could send them in batches below the keepalive limit, but that's
> not what serf does. serf will write up to the discovered keep-alive
> limit or until we get EAGAIN then write up to keep-alive limit when
> the socket becomes writeable again.
>
Right, but is there any reason for the server not to allow an unlimited
number of requests per connection rather than hanging up after a limit?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Status of ra_serf
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On 2/18/06, Phillip Susi <ps...@cfl.rr.com> wrote:
> I see. Then yes, it makes sense to limit the depth of the pipeline. My
> point though, wasn't that you should send 5000 requests at once, but
> that there is no reason you can't send them in batches of say, 25, and
> just keep sending more as the first requests complete, keeping the
> pipeline full the whole time. Why should the server close the socket
> and require you to reconnect to send another batch of requests?
You could send them in batches below the keepalive limit, but that's
not what serf does. serf will write up to the discovered keep-alive
limit or until we get EAGAIN then write up to keep-alive limit when
the socket becomes writeable again.
> > In order to stick the full-text in the XML response to the REPORT for
> > ra_dav, mod_dav_svn base-64 encodes every file such that it will be
> > valid XML parsable by expat. Base-64 encoding is moderately expensive
> > and increases the overall space; however, using mod_deflate lets us
> > recover some of the space at the cost of even more CPU.
> >
>
> Outch. That kind of sucks. I wonder if the REPORT couldn't just refer
> you to the files you could fetch in binary with GET rather than embed
> them in the XML response.
Yes, that's the strategy that serf is taking. Neon can't or rather it
used to, but the performance was abysmal since neon couldn't do async
or pipelining. The only thing neon could do efficiently was just
issue one large REPORT request. -- justin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Status of ra_serf
Posted by Phillip Susi <ps...@cfl.rr.com>.
Justin Erenkrantz wrote:
> No. Serf allocates a pool to the request when it decides to deliver
> it. If we decided to not use a pool strategy, it would make my life
> (and any one coding at serf's level) suck. So, that means each
> request requires 8KB of memory at a minimum. For 1000 requests,
> that's 8KB*1000 = ~8MB. If we remove a pipeline limit and write 5000
> requests, we're talking ~40MB of overhead. Feasible, but there's a
> steep cost for memory allocation - reducing our memory footprint makes
> us faster than if we tried to be cute and write 5000 requests when we
> know that we're not going to need them all. (Also, by delaying our
> memory allocation, we get the benefits of pools such that we stabilize
> our memory usage very quickly and it plateaus.)
>
I see. Then yes, it makes sense to limit the depth of the pipeline. My
point though, wasn't that you should send 5000 requests at once, but
that there is no reason you can't send them in batches of say, 25, and
just keep sending more as the first requests complete, keeping the
pipeline full the whole time. Why should the server close the socket
and require you to reconnect to send another batch of requests?
> In order to stick the full-text in the XML response to the REPORT for
> ra_dav, mod_dav_svn base-64 encodes every file such that it will be
> valid XML parsable by expat. Base-64 encoding is moderately expensive
> and increases the overall space; however, using mod_deflate lets us
> recover some of the space at the cost of even more CPU.
>
Outch. That kind of sucks. I wonder if the REPORT couldn't just refer
you to the files you could fetch in binary with GET rather than embed
them in the XML response.
>> Also to be clear, you are saying that the increased multiple extra get
>> connections increase the load on the server, but that increase is
>> negligible compared to the load from the report request?
>
> Correct.
>
>> I think that in the vast majority of cases, the bottleneck is going to
>> be the network, not the local disk. Most hard disks these days can
>> handle 10 MB/s easily, but not many people have >= 100 Mbps connections
>> to a svn server. Given that, splitting the data stream into multiple
>> tcp sockets tends to lower the total throughput due to the competition,
>> which will increase overall checkout times.
>
> We'll need to let the data trump any hypotheses. =) -- justin
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Status of ra_serf
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On 2/17/06, Phillip Susi <ps...@cfl.rr.com> wrote:
> I don't understand. Presumably you already have a list of files you
> want to fetch in memory. That memory is allocated anyhow, no matter if
> you are sending the request list or not, so why would take any more
> memory to start sending the requests to the server? I would think you
> would just start walking the list of files and write GET requests to the
> socket as long as the socket still has buffer space. If that's the case
> then the only extra memory uses is the kernel socket send buffer, which
> should be limited to a reasonable value. You're going to use the same
> amount of memory whether you write 100 get requests to one socket, or 25
> to each of 4.
No. Serf allocates a pool to the request when it decides to deliver
it. If we decided to not use a pool strategy, it would make my life
(and any one coding at serf's level) suck. So, that means each
request requires 8KB of memory at a minimum. For 1000 requests,
that's 8KB*1000 = ~8MB. If we remove a pipeline limit and write 5000
requests, we're talking ~40MB of overhead. Feasible, but there's a
steep cost for memory allocation - reducing our memory footprint makes
us faster than if we tried to be cute and write 5000 requests when we
know that we're not going to need them all. (Also, by delaying our
memory allocation, we get the benefits of pools such that we stabilize
our memory usage very quickly and it plateaus.)
> >> True, but doesn't quadrupling the connections to the server increase the
> >> load it on quite a bit? That's what I'm concerned about.
> >
> > No, it doesn't - compared to the large REPORT response ra_dav fetches.
> > The server isn't having to do any base-64 encoding - that's the real
> > room for opportunity that ra_serf has as we're just not going to be
> > more optimal on the number of connections. Therefore, breaking the
> > activity into smaller chunks helps ease the computational load on the
> > server (provided the disk I/O can keep up).
> >
>
> Hrm... you got me curious now... what's using base-64 encoding and why?
In order to stick the full-text in the XML response to the REPORT for
ra_dav, mod_dav_svn base-64 encodes every file such that it will be
valid XML parsable by expat. Base-64 encoding is moderately expensive
and increases the overall space; however, using mod_deflate lets us
recover some of the space at the cost of even more CPU.
> Also to be clear, you are saying that the increased multiple extra get
> connections increase the load on the server, but that increase is
> negligible compared to the load from the report request?
Correct.
> I think that in the vast majority of cases, the bottleneck is going to
> be the network, not the local disk. Most hard disks these days can
> handle 10 MB/s easily, but not many people have >= 100 Mbps connections
> to a svn server. Given that, splitting the data stream into multiple
> tcp sockets tends to lower the total throughput due to the competition,
> which will increase overall checkout times.
We'll need to let the data trump any hypotheses. =) -- justin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Status of ra_serf
Posted by Phillip Susi <ps...@cfl.rr.com>.
Justin Erenkrantz wrote:
>
> Remember that we may able to truly parallelize the requests if the
> server has multiple CPUs, etc.
>
> Also, if we try to 'flood' the server with an unlimited pipeline
> depth, we'll take up more memory on the client-side than needed as we
> have to 'manage' more concurrent requests. Some recent commits to
> serf changed the API to only allocate memory when we're about to write
> the request not when we create the request. It cut the memory
> consumption by half.
>
I don't understand. Presumably you already have a list of files you
want to fetch in memory. That memory is allocated anyhow, no matter if
you are sending the request list or not, so why would take any more
memory to start sending the requests to the server? I would think you
would just start walking the list of files and write GET requests to the
socket as long as the socket still has buffer space. If that's the case
then the only extra memory uses is the kernel socket send buffer, which
should be limited to a reasonable value. You're going to use the same
amount of memory whether you write 100 get requests to one socket, or 25
to each of 4.
>> True, but doesn't quadrupling the connections to the server increase the
>> load it on quite a bit? That's what I'm concerned about.
>
> No, it doesn't - compared to the large REPORT response ra_dav fetches.
> The server isn't having to do any base-64 encoding - that's the real
> room for opportunity that ra_serf has as we're just not going to be
> more optimal on the number of connections. Therefore, breaking the
> activity into smaller chunks helps ease the computational load on the
> server (provided the disk I/O can keep up).
>
Hrm... you got me curious now... what's using base-64 encoding and why?
Also to be clear, you are saying that the increased multiple extra get
connections increase the load on the server, but that increase is
negligible compared to the load from the report request?
>> It also isn't
>> as helpful for checkout times as keeping one ( or two if you know the
>> first will be blocked for a while doing processing on the server )
>> connections pipelined.
>
> Again, it's a function of what resources the server has to offer.
> Ideally, we'd like to make the bottleneck the *client* disk I/O not
> the network or the server disk I/O.
>
I think that in the vast majority of cases, the bottleneck is going to
be the network, not the local disk. Most hard disks these days can
handle 10 MB/s easily, but not many people have >= 100 Mbps connections
to a svn server. Given that, splitting the data stream into multiple
tcp sockets tends to lower the total throughput due to the competition,
which will increase overall checkout times.
>> Yes, the server will hang up after x requests, but when you are issuing
>> x requests at the same time on 4 connections, they will compete with
>> each other and prevent their tcp windows from opening fully. For very
>> low values of x ( 1-10 ) then 4 connections might give an improvement
>> because they are all so short lived anyhow that they can't reach full
>> open windows, and you get rid of the reconnect latency. For values of x
>> >= 100 though, I think 2 connections would give better results. Should
>> make an interesting test...
>
> There's a distinct need to profile our behaviors to ensure we're being
> as optimal as we can make it. ;-) -- justin
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Status of ra_serf
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On 2/17/06, Phillip Susi <ps...@cfl.rr.com> wrote:
> Yes, but for years web browsers have been rather dumb and opened 4
> connections because they only issue one request per connection. Beefy
> or not, servers don't appreciate having to handle 4000 simultaneous
> connections when there are only 1000 clients.
They certainly do keepalive but not pipelining. Firefox has long had
pipelining as a config option, but there's *just* enough quirkiness to
not have it on by default for a general purpose HTTP client. ra_serf
doesn't need to talk to every web server in the universe - so we can
assume that pipelining more or less will work. (Some proxies may get
in the way, sure.)
> > Right. I don't know what the 'best' value is yet. I think it's over
> > 100 (absolutely over 10!), but I'm beginning to think 1000 might be
> > too high. One of the output of ra_serf may very well be, "Hey, you
> > server admins, tune your Apache config with these values."
> >
>
> Is there any reason not to leave it completely unlimited? I can't see
> one. Why kick the client off and make them reconnect for no good reason?
Remember that we may able to truly parallelize the requests if the
server has multiple CPUs, etc.
Also, if we try to 'flood' the server with an unlimited pipeline
depth, we'll take up more memory on the client-side than needed as we
have to 'manage' more concurrent requests. Some recent commits to
serf changed the API to only allocate memory when we're about to write
the request not when we create the request. It cut the memory
consumption by half.
> True, but doesn't quadrupling the connections to the server increase the
> load it on quite a bit? That's what I'm concerned about.
No, it doesn't - compared to the large REPORT response ra_dav fetches.
The server isn't having to do any base-64 encoding - that's the real
room for opportunity that ra_serf has as we're just not going to be
more optimal on the number of connections. Therefore, breaking the
activity into smaller chunks helps ease the computational load on the
server (provided the disk I/O can keep up).
> It also isn't
> as helpful for checkout times as keeping one ( or two if you know the
> first will be blocked for a while doing processing on the server )
> connections pipelined.
Again, it's a function of what resources the server has to offer.
Ideally, we'd like to make the bottleneck the *client* disk I/O not
the network or the server disk I/O.
> Yes, the server will hang up after x requests, but when you are issuing
> x requests at the same time on 4 connections, they will compete with
> each other and prevent their tcp windows from opening fully. For very
> low values of x ( 1-10 ) then 4 connections might give an improvement
> because they are all so short lived anyhow that they can't reach full
> open windows, and you get rid of the reconnect latency. For values of x
> >= 100 though, I think 2 connections would give better results. Should
> make an interesting test...
There's a distinct need to profile our behaviors to ensure we're being
as optimal as we can make it. ;-) -- justin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Status of ra_serf
Posted by Michael Sweet <mi...@easysw.com>.
Phillip Susi wrote:
> ...
> Yes, the server will hang up after x requests, but when you are issuing
> x requests at the same time on 4 connections, they will compete with
> each other and prevent their tcp windows from opening fully. For very
> low values of x ( 1-10 ) then 4 connections might give an improvement
> because they are all so short lived anyhow that they can't reach full
> open windows, and you get rid of the reconnect latency. For values of x
> >= 100 though, I think 2 connections would give better results. Should
> make an interesting test...
My $0.02: not having looked at the implementation yet, it seems like
you might benefit from starting with 1 connection to get the report,
and then opening new connections to the server only as needed to
satisfy a new request (basically the reverse of what Apache itself
does when accepting new client connections and firing up new server
instances as needed).
If you have an "idle" connection (or one that is almost finished),
you can just use that connection to request the next bit of data
rather than opening up another connection to the server. If you
don't have any idle connections, open a new one up to a configured
limit (4 by default), and then schedule the request on the least
busy connection. Somewhat complex, but not overly so, and it will
adapt to the network/server conditions automatically to give the
best overall performance.
--
______________________________________________________________________
Michael Sweet, Easy Software Products mike at easysw dot com
Internet Printing and Publishing Software http://www.easysw.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Status of ra_serf
Posted by Phillip Susi <ps...@cfl.rr.com>.
Justin Erenkrantz wrote:
> For those really busy servers, they are likely a bit beefier than the
> client. For example, svn.apache.org's repository is on a RAID-5'd
> dual-processor machine sitting on a big fat pipe. If we were that
> gun-shy, the client would be sitting idle when there's no need to do
> so as the server can keep up.
>
> I'm not personally worried about opening up 4 concurrent connections.
> Again, that has been the common strategy of every web browser for
> years and is a decent precedent for us to follow. (In return, we're
> doing much simpler requests.)
>
Yes, but for years web browsers have been rather dumb and opened 4
connections because they only issue one request per connection. Beefy
or not, servers don't appreciate having to handle 4000 simultaneous
connections when there are only 1000 clients.
> Right. I don't know what the 'best' value is yet. I think it's over
> 100 (absolutely over 10!), but I'm beginning to think 1000 might be
> too high. One of the output of ra_serf may very well be, "Hey, you
> server admins, tune your Apache config with these values."
>
Is there any reason not to leave it completely unlimited? I can't see
one. Why kick the client off and make them reconnect for no good reason?
> At the very least, I'll ensure that svn.apache.org (and hopefully
> svn.collab.net) are tuned for ra_serf. So, I'll be personally happy.
> ;-)
>
> My short-term goal with ra_serf is not to focus on server-side
> changes. I want ra_serf to work with any 1.0+ server. If we can
> later add code to optimize the server, all the better. But, ra_serf
> will have zero appeal if only works against 'newer' Subversion
> servers.
>
True.
> This also keys the performance goals: ra_serf should be competitive in
> most cases with ra_dav or no one will be interested in using it. If
> ra_serf is a couple of percentage points behind from ra_dav but the
> server load drops by half, then that might be enough to convince folks
> to switch our default to ra_serf. But, an average performance penalty
> of more than a few percentage points is going to be a showstopper.
> Certainly, if we can beat ra_dav in most cases, it'll be really easy
> to convince people to switch. ;-)
>
True, but doesn't quadrupling the connections to the server increase the
load it on quite a bit? That's what I'm concerned about. It also isn't
as helpful for checkout times as keeping one ( or two if you know the
first will be blocked for a while doing processing on the server )
connections pipelined.
<snip>
> Again, the 'fetching' connection won't live very long in the default
> case. That's why we have to multiplex connections. -- justin
>
Yes, the server will hang up after x requests, but when you are issuing
x requests at the same time on 4 connections, they will compete with
each other and prevent their tcp windows from opening fully. For very
low values of x ( 1-10 ) then 4 connections might give an improvement
because they are all so short lived anyhow that they can't reach full
open windows, and you get rid of the reconnect latency. For values of x
>= 100 though, I think 2 connections would give better results. Should
make an interesting test...
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Status of ra_serf
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On 2/17/06, Phillip Susi <ps...@cfl.rr.com> wrote:
> Ideally you want to open one connection and send the 100 requests to
> prime the pipeline. Then JUST as the fist connection finishes with the
> last file, you want the next 100 requests to hit the server on the other
> connection. Having two connections each pulling 100 requests at the
> same time causes disk thrashing and network packet collisions, which you
> want to avoid.
For those really busy servers, they are likely a bit beefier than the
client. For example, svn.apache.org's repository is on a RAID-5'd
dual-processor machine sitting on a big fat pipe. If we were that
gun-shy, the client would be sitting idle when there's no need to do
so as the server can keep up.
I'm not personally worried about opening up 4 concurrent connections.
Again, that has been the common strategy of every web browser for
years and is a decent precedent for us to follow. (In return, we're
doing much simpler requests.)
> It would also be nice if the server would just raise that stupid cap so
> you don't have to waste time building a new TCP connection and getting
> the window open.
Right. I don't know what the 'best' value is yet. I think it's over
100 (absolutely over 10!), but I'm beginning to think 1000 might be
too high. One of the output of ra_serf may very well be, "Hey, you
server admins, tune your Apache config with these values."
At the very least, I'll ensure that svn.apache.org (and hopefully
svn.collab.net) are tuned for ra_serf. So, I'll be personally happy.
;-)
> > The problem is that the client is sitting around idle until the server
> > finishes the REPORT response. The REPORT takes a *long* time to
> > completely generate as there's a lot of data that it is sending back
> > (the entire list of files to fetch + some properties for each file).
> > (AFAICT, REPORT is largely constrained by the backend repository I/O
> > speed.) Also, mod_deflate and mod_dav_svn interact in such a way that
> > for REPORT requests that don't include full-text, it will be buffered
> > on the server until the REPORT response is completed (this is why
> > ra_serf won't request gzip compression on the REPORT - yes, this is a
> > nasty server-side bug; I would like to track it down,
> > time-permitting).
> >
>
> I would ask why does it take so long to generate the report? Maybe that
> has some room for improvement. Given that the report request takes so
> long to generate, and during that time the connection is blocked, yes,
> it would be a good idea to open another connection to download files
> that are more readily available.
My short-term goal with ra_serf is not to focus on server-side
changes. I want ra_serf to work with any 1.0+ server. If we can
later add code to optimize the server, all the better. But, ra_serf
will have zero appeal if only works against 'newer' Subversion
servers.
This also keys the performance goals: ra_serf should be competitive in
most cases with ra_dav or no one will be interested in using it. If
ra_serf is a couple of percentage points behind from ra_dav but the
server load drops by half, then that might be enough to convince folks
to switch our default to ra_serf. But, an average performance penalty
of more than a few percentage points is going to be a showstopper.
Certainly, if we can beat ra_dav in most cases, it'll be really easy
to convince people to switch. ;-)
Now, ra_dav is using really sneaky tricks to be as fast as possible:
one large REPORT isn't doing too badly in certain setups. There's a
fairly high bar here. I do think ra_serf is doing better than ra_dav
was before it switched to an update-report; so serf is allowing us to
go back to the original checkout techniques we used way back when
(before 0.25.x, I think) and get within shouting distance of ra_dav's
current 'bastardized' techniques for checkouts.
> You shouldn't need more than one extra connection though, and ideally it
> would be great if you could ask the server to begin generating the
> report in the background and spool it in a temp file, then download
> other files you know you need, THEN fetch the report from the temp file.
> That way you wouldn't need the extra connection.
Async responses aren't part of HTTP. It'd be a part of Waka though. ;-)
> > The real trick here is that there is no reason to wait for the REPORT
> > response to finish to start acquiring the data we already know we
> > need. There's no real I/O happening on the client-side - therefore,
> > it is more efficient to open a new connection to the server and start
> > pulling down the files as soon as we know we need it.
> >
>
> Aye... but again, one extra connection should be sufficient since only
> the report connection is blocked; the other connection can be kept
> pipelined.
Again, the 'fetching' connection won't live very long in the default
case. That's why we have to multiplex connections. -- justin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Status of ra_serf
Posted by kf...@collab.net.
Phillip Susi <ps...@cfl.rr.com> writes:
> I would ask why does it take so long to generate the report? Maybe
> that has some room for improvement.
There's always room for improvement, but note that the length of time
required to generate the full report is, in the end, bound to the size
and complexity of what's being reported. Say you're doing an update
from r1 to r8973468, on a system with complex authz requirements.
Now, that's just going to take time, because of the nature of the
data. It can only be improved up to a certain limit.
-Karl
--
www.collab.net <> CollabNet | Distributed Development On Demand
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Status of ra_serf
Posted by Phillip Susi <ps...@cfl.rr.com>.
Justin Erenkrantz wrote:
> No. The problem with HTTP connections is that they have a very finite
> lifetime. The default configuration for Apache httpd is to only allow
> 100 requests on a connection then close it. This is tunable by the
> server admins and one of the factors we'll have to analyze is "what's
> the optimal number of requests on a connection?" We currently issue
> two HTTP requests for each file we request (a PROPFIND and GET). So,
> that means, in the default config, we can only get 50 files per TCP
> connection out of an out-of-the-box configuration. (Bumping it up to
> 1000 greatly increases the pipeline depth which increases the memory
> usage on the client as well.)
>
> Therefore, since the connections can go away, what we ideally want to
> do is stagger our requests across connections such that the network is
> always active and our 'pipeline' is full: this is why more than one
> connection is needed for maximum efficiency.
>
Ideally you want to open one connection and send the 100 requests to
prime the pipeline. Then JUST as the fist connection finishes with the
last file, you want the next 100 requests to hit the server on the other
connection. Having two connections each pulling 100 requests at the
same time causes disk thrashing and network packet collisions, which you
want to avoid.
It would also be nice if the server would just raise that stupid cap so
you don't have to waste time building a new TCP connection and getting
the window open.
> The problem is that the client is sitting around idle until the server
> finishes the REPORT response. The REPORT takes a *long* time to
> completely generate as there's a lot of data that it is sending back
> (the entire list of files to fetch + some properties for each file).
> (AFAICT, REPORT is largely constrained by the backend repository I/O
> speed.) Also, mod_deflate and mod_dav_svn interact in such a way that
> for REPORT requests that don't include full-text, it will be buffered
> on the server until the REPORT response is completed (this is why
> ra_serf won't request gzip compression on the REPORT - yes, this is a
> nasty server-side bug; I would like to track it down,
> time-permitting).
>
I would ask why does it take so long to generate the report? Maybe that
has some room for improvement. Given that the report request takes so
long to generate, and during that time the connection is blocked, yes,
it would be a good idea to open another connection to download files
that are more readily available.
You shouldn't need more than one extra connection though, and ideally it
would be great if you could ask the server to begin generating the
report in the background and spool it in a temp file, then download
other files you know you need, THEN fetch the report from the temp file.
That way you wouldn't need the extra connection.
> The real trick here is that there is no reason to wait for the REPORT
> response to finish to start acquiring the data we already know we
> need. There's no real I/O happening on the client-side - therefore,
> it is more efficient to open a new connection to the server and start
> pulling down the files as soon as we know we need it.
>
Aye... but again, one extra connection should be sufficient since only
the report connection is blocked; the other connection can be kept
pipelined.
> To put numbers behind it, for a checkout of an httpd 2.2.0 tarball, it
> takes roughly 7 seconds from the time the server starts writing the
> REPORT response until it is completely finished (with the client just
> saying 'yah, whatever' and ignoring it). If we maintain just one
> connection, ra_serf will indeed queue all of them up - but the server
> can't begin to respond until it's done generating and writing the
> REPORT.
>
> Without multiple connections, a checkout with ra_serf would take about
> 35 seconds. With multiple connections, we can currently do that same
> checkout in under 25 seconds. In some cases, the speed advantage is a
> factor of 2 or more by acquiring files as soon as we know we need
> them.
>
> HTH. -- justin
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Status of ra_serf
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On 2/17/06, Greg Hudson <gh...@mit.edu> wrote:
> On Fri, 2006-02-17 at 02:26 -0800, Justin Erenkrantz wrote:
> > As of r18500, we now open 4 connections to the server in order to do
> > the checkout. Part of the reason for this is that as we parse the
> > REPORT response, we can start to fetch the actual content. While we
> > can keep it to one connection (and have done so until just now), we'll
> > essentially be blocked for the length of the REPORT response
>
> Aren't two connections good enough for maximum efficiency, then? If
> you're pipelining, I don't see why three connections for the GETs is
> better than one.
No. The problem with HTTP connections is that they have a very finite
lifetime. The default configuration for Apache httpd is to only allow
100 requests on a connection then close it. This is tunable by the
server admins and one of the factors we'll have to analyze is "what's
the optimal number of requests on a connection?" We currently issue
two HTTP requests for each file we request (a PROPFIND and GET). So,
that means, in the default config, we can only get 50 files per TCP
connection out of an out-of-the-box configuration. (Bumping it up to
1000 greatly increases the pipeline depth which increases the memory
usage on the client as well.)
Therefore, since the connections can go away, what we ideally want to
do is stagger our requests across connections such that the network is
always active and our 'pipeline' is full: this is why more than one
connection is needed for maximum efficiency.
> I'm actually unsure why you need more than one connection total. You
> can be sending GETs to the server as you're parsing the REPORT response,
> then start handling GET responses as soon as the REPORT response is
> done. The server should be sending data full-time, with only one
> round-trip delay at the beginning to send off the first GET request.
The problem is that the client is sitting around idle until the server
finishes the REPORT response. The REPORT takes a *long* time to
completely generate as there's a lot of data that it is sending back
(the entire list of files to fetch + some properties for each file).
(AFAICT, REPORT is largely constrained by the backend repository I/O
speed.) Also, mod_deflate and mod_dav_svn interact in such a way that
for REPORT requests that don't include full-text, it will be buffered
on the server until the REPORT response is completed (this is why
ra_serf won't request gzip compression on the REPORT - yes, this is a
nasty server-side bug; I would like to track it down,
time-permitting).
The real trick here is that there is no reason to wait for the REPORT
response to finish to start acquiring the data we already know we
need. There's no real I/O happening on the client-side - therefore,
it is more efficient to open a new connection to the server and start
pulling down the files as soon as we know we need it.
To put numbers behind it, for a checkout of an httpd 2.2.0 tarball, it
takes roughly 7 seconds from the time the server starts writing the
REPORT response until it is completely finished (with the client just
saying 'yah, whatever' and ignoring it). If we maintain just one
connection, ra_serf will indeed queue all of them up - but the server
can't begin to respond until it's done generating and writing the
REPORT.
Without multiple connections, a checkout with ra_serf would take about
35 seconds. With multiple connections, we can currently do that same
checkout in under 25 seconds. In some cases, the speed advantage is a
factor of 2 or more by acquiring files as soon as we know we need
them.
HTH. -- justin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Status of ra_serf
Posted by Phillip Susi <ps...@cfl.rr.com>.
Greg Stein wrote:
> just fyi, Phillip: Justin is *very* aware of this, and many of the
> other details you mention in your email. He is a long-time contributor
> to Apache, and one of the design goals of serf is to take advantage of
> pipelining (which nearly all HTTP clients do *NOT* do). Point is:
> rejigger your emails to assume Justin knows HTTP, and ask why he is
> making certain choices, instead of teaching him about HTTP :-)
>
Cool, thanks for the heads up. I see that now that I'm reading the
other messages in this thread. I too hope to see a day when the clients
utilize pipelining and the server takes advantage of zero copy async IO
with a number of threads/processes significantly less than the number of
clients.
> The server closes it down periodically. Thus, the client needs to
> reopen it. To keep traffic flowing, you want to have another
> connection ready for immediate use while you go thru the TCP setup to
> reopen the closed connection.
>
Right... you want to open the next connection and prime it's pipeline
just before the first connection is closed. It sounded like ra_serf
just opens 4 connections and gets them all going in parallel. Maybe I
just misunderstood?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Status of ra_serf
Posted by Greg Stein <gs...@lyra.org>.
On Fri, Feb 17, 2006 at 12:02:52PM -0500, Phillip Susi wrote:
> Justin Erenkrantz wrote:
> >
> >Again, HTTP connections are ideally stateless - so, we can't have a
> >very deep pipeline before the server decides to kill us off.
>
> Most servers these days allow you to issue a number of requests over the
> same connection.
just fyi, Phillip: Justin is *very* aware of this, and many of the
other details you mention in your email. He is a long-time contributor
to Apache, and one of the design goals of serf is to take advantage of
pipelining (which nearly all HTTP clients do *NOT* do). Point is:
rejigger your emails to assume Justin knows HTTP, and ask why he is
making certain choices, instead of teaching him about HTTP :-)
>...
> >If you kept it at one connection, you'd have lengthy delays where
> >there is no network traffic because you're constantly reopening the
> >TCP sockets. This isn't ra_svn which assumes a persistent TCP
> >connection for the lifetime of the RA session. -- justin
>
> You aren't reopening the socket if you keep it open and pipelined, thus
> there are no delays. HTTP/1.1 allows persistent connections.
The server closes it down periodically. Thus, the client needs to
reopen it. To keep traffic flowing, you want to have another
connection ready for immediate use while you go thru the TCP setup to
reopen the closed connection.
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Status of ra_serf
Posted by Phillip Susi <ps...@cfl.rr.com>.
Justin Erenkrantz wrote:
>
> Again, HTTP connections are ideally stateless - so, we can't have a
> very deep pipeline before the server decides to kill us off.
Most servers these days allow you to issue a number of requests over the
same connection. IIRC, you can issue the request for the next document
while the first document is still streaming down at you. That will keep
the connection pipelined and result in optimum performance since the TCP
windows will open up and stream as fast as the network allows, with no
downtime, and no competition among multiple connections.
>
> The most optimum solution from the client's perspective is to open
> #-of-files-to-get/pipeline-depth connections to the server. That's
> unrealistic, so we cap ourselves to 4. =)
>
The most optimum solution from the client's perspective is to keep data
moving with no down time. A single pipelined connection will do this,
and do it better than multiple connections which compete with each other
and may never get a fully open window.
> If you kept it at one connection, you'd have lengthy delays where
> there is no network traffic because you're constantly reopening the
> TCP sockets. This isn't ra_svn which assumes a persistent TCP
> connection for the lifetime of the RA session. -- justin
>
You aren't reopening the socket if you keep it open and pipelined, thus
there are no delays. HTTP/1.1 allows persistent connections.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Status of ra_serf
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On 2/17/06, Phillip Susi <ps...@cfl.rr.com> wrote:
> Aye, I was wondering the same thing. As long as you keep the
> connection(s) pipelined so they are always pushing data as fast as they
> can, then one is better than many since the many will compete with each
Again, HTTP connections are ideally stateless - so, we can't have a
very deep pipeline before the server decides to kill us off.
The most optimum solution from the client's perspective is to open
#-of-files-to-get/pipeline-depth connections to the server. That's
unrealistic, so we cap ourselves to 4. =)
> Seeing as how the one test done on a local server that isn't sharing
> bandwidth with other users turned out to be slower than ra_dav, that
> tends to support my theory. A single connection is more efficient, but
> using multiple connections on a heavily used server enables you to steal
> more bandwidth from other users. This is not a very good idea. If you
> start using 4 connections to steal more bandwidth, then someone else is
> going to start using 5, then 6, and so on. Eventually the server will
> be hit with so many connections that at best, a lot of collisions will
> happen and everyone's throughput will drop, and at worst, the server
> will take a dump from the overload.
If you kept it at one connection, you'd have lengthy delays where
there is no network traffic because you're constantly reopening the
TCP sockets. This isn't ra_svn which assumes a persistent TCP
connection for the lifetime of the RA session. -- justin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Status of ra_serf
Posted by Phillip Susi <ps...@cfl.rr.com>.
Greg Hudson wrote:
>
> Aren't two connections good enough for maximum efficiency, then? If
> you're pipelining, I don't see why three connections for the GETs is
> better than one.
>
Aye, I was wondering the same thing. As long as you keep the
connection(s) pipelined so they are always pushing data as fast as they
can, then one is better than many since the many will compete with each
other and slow themselves down. The only way this would be otherwise is
if you are talking to a busy server whose upstream is saturated, in
which case having 4 of your own connections will steal more bandwidth
from other users, though that's really an unfair and inefficient setup.
> I'm actually unsure why you need more than one connection total. You
> can be sending GETs to the server as you're parsing the REPORT response,
> then start handling GET responses as soon as the REPORT response is
> done. The server should be sending data full-time, with only one
> round-trip delay at the beginning to send off the first GET request.
>
Exactly.
Seeing as how the one test done on a local server that isn't sharing
bandwidth with other users turned out to be slower than ra_dav, that
tends to support my theory. A single connection is more efficient, but
using multiple connections on a heavily used server enables you to steal
more bandwidth from other users. This is not a very good idea. If you
start using 4 connections to steal more bandwidth, then someone else is
going to start using 5, then 6, and so on. Eventually the server will
be hit with so many connections that at best, a lot of collisions will
happen and everyone's throughput will drop, and at worst, the server
will take a dump from the overload.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Status of ra_serf
Posted by Greg Hudson <gh...@MIT.EDU>.
On Fri, 2006-02-17 at 02:26 -0800, Justin Erenkrantz wrote:
> As of r18500, we now open 4 connections to the server in order to do
> the checkout. Part of the reason for this is that as we parse the
> REPORT response, we can start to fetch the actual content. While we
> can keep it to one connection (and have done so until just now), we'll
> essentially be blocked for the length of the REPORT response
Aren't two connections good enough for maximum efficiency, then? If
you're pipelining, I don't see why three connections for the GETs is
better than one.
I'm actually unsure why you need more than one connection total. You
can be sending GETs to the server as you're parsing the REPORT response,
then start handling GET responses as soon as the REPORT response is
done. The server should be sending data full-time, with only one
round-trip delay at the beginning to send off the first GET request.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org