You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@subversion.apache.org by st...@apache.org on 2012/09/20 13:41:26 UTC
svn commit: r1387981 - /subversion/branches/10Gb/BRANCH-README
Author: stefan2
Date: Thu Sep 20 11:41:25 2012
New Revision: 1387981
URL: http://svn.apache.org/viewvc?rev=1387981&view=rev
Log:
On 10Gb branch, add a README.
* BRANCH-README: new file
Added:
subversion/branches/10Gb/BRANCH-README
Added: subversion/branches/10Gb/BRANCH-README
URL: http://svn.apache.org/viewvc/subversion/branches/10Gb/BRANCH-README?rev=1387981&view=auto
==============================================================================
--- subversion/branches/10Gb/BRANCH-README (added)
+++ subversion/branches/10Gb/BRANCH-README Thu Sep 20 11:41:25 2012
@@ -0,0 +1,15 @@
+This branch will receive a number of patches that boost
+svn:// single-threaded throughput from about 5Gb/s to
+10Gb/s for typical source code, i.e. becomes capable of
+saturating a 10Gb connection.
+
+Bottlenecks to address:
+
+* frequent cancellation checks (intense OS interaction)
+* in-memory copies (membuffer cache -> empty deltification
+ -> output buffer -> network stack)
+* various CPU-heavy tasks
+
+The patches have been written quite some time ago and I
+want them off my disk. OTOH, release 1.8 shall not be
+endangered.
Re: svn commit: r1387981 - /subversion/branches/10Gb/BRANCH-README
Posted by Johan Corveleyn <jc...@gmail.com>.
On Thu, Sep 20, 2012 at 1:41 PM, <st...@apache.org> wrote:
> Author: stefan2
> Date: Thu Sep 20 11:41:25 2012
> New Revision: 1387981
>
> URL: http://svn.apache.org/viewvc?rev=1387981&view=rev
> Log:
> On 10Gb branch, add a README.
>
> * BRANCH-README: new file
>
> Added:
> subversion/branches/10Gb/BRANCH-README
>
> Added: subversion/branches/10Gb/BRANCH-README
> URL: http://svn.apache.org/viewvc/subversion/branches/10Gb/BRANCH-README?rev=1387981&view=auto
> ==============================================================================
> --- subversion/branches/10Gb/BRANCH-README (added)
> +++ subversion/branches/10Gb/BRANCH-README Thu Sep 20 11:41:25 2012
> @@ -0,0 +1,15 @@
> +This branch will receive a number of patches that boost
> +svn:// single-threaded throughput from about 5Gb/s to
> +10Gb/s for typical source code, i.e. becomes capable of
> +saturating a 10Gb connection.
> +
> +Bottlenecks to address:
> +
> +* frequent cancellation checks (intense OS interaction)
> +* in-memory copies (membuffer cache -> empty deltification
> + -> output buffer -> network stack)
> +* various CPU-heavy tasks
> +
> +The patches have been written quite some time ago and I
> +want them off my disk. OTOH, release 1.8 shall not be
> +endangered.
As a mod_dav_svn user, I'm wondering:
- How much of your current set of patches will also benefit
(indirectly perhaps) mod_dav_svn?
- For the things that only help svnserve, how much of those could be
implemented for mod_dav_svn as well (practically and/or theoretically
-- perhaps some of those would need a totally different approach, but
maybe the principle is valid for mod_dav_svn as well)?
Or is mod_dav_svn just not able to get a higher throughput because of
other reasons (protocol overhead, other bottlenecks, ...)?
--
Johan
Re: svn commit: r1387981 - /subversion/branches/10Gb/BRANCH-README
Posted by Stefan Fuhrmann <st...@wandisco.com>.
On Fri, Sep 21, 2012 at 3:51 AM, Branko Čibej <br...@wandisco.com> wrote:
> On 20.09.2012 23:35, Stefan Fuhrmann wrote:
> > On Thu, Sep 20, 2012 at 9:22 PM, Branko Čibej <br...@wandisco.com>
> wrote:
> >
> >> On 20.09.2012 13:41, stefan2@apache.org wrote:
> >>> +Bottlenecks to address:
> >>> +
> >>> +* frequent cancellation checks (intense OS interaction)
> >> I don't understand this part: since when do cancellation checks involve
> >> interaction with the OS? I thought the cancellation flag was an (atomic,
> >> volatile) thing that requires neither cpu-level atomic operations, nor
> >> higher-level locking.
> >>
> > It may not actually be "the OS" but the phenomenon is real.
> > check_for_error() in ra_svn/editorp.c calls apr_poll(), which in
> > turn calls select().
> >
> > To make things worse, we buffer small output data like props
> > etc. but check for errors after each of them. So, more select()
> > calls then socket writes ...
>
> OK, so this is actually the server looking for abort commands from the
> client, which is a bit different than the client checking the
> cancellation flag. :)
>
Yup. I changed the lingo in a few places to make that distinction
more clear.
> > My idea is to short-circuit the check until at least x amount
> > of data has been sent.
>
> I'd expect that the same select/poll that waits for the write end of the
> socket to become ready can also notice that the read end has data
> available; making the whole thing asynchronous. Blindly sending data
> after the client aborted the edit seems like the wrong way to solve
> throughput issues.
>
We *do* blindly send data - up to a point since outgoing traffic
is being collected in an ra_svn-internal TX buffer. So, if some
of it would "offend" the client, the remainder of that buffer will
still be sent unconditionally.
> Of course, taking a quick look at the implementation, this might involve
> rewriting all of ra_svn's network layer ...
>
The whole communication is already asynchronous. It is simply
a cutesy to stop sending data before we get told by the OS that
the client closed its part of the connection anyway.
-- Stefan^2.
--
*
Join us this October at Subversion Live
2012<http://www.wandisco.com/svn-live-2012>
for two days of best practice SVN training, networking, live demos,
committer meet and greet, and more! Space is limited, so get signed up
today<http://www.wandisco.com/svn-live-2012>
!
*
Re: svn commit: r1387981 - /subversion/branches/10Gb/BRANCH-README
Posted by Branko Čibej <br...@wandisco.com>.
On 20.09.2012 23:35, Stefan Fuhrmann wrote:
> On Thu, Sep 20, 2012 at 9:22 PM, Branko Čibej <br...@wandisco.com> wrote:
>
>> On 20.09.2012 13:41, stefan2@apache.org wrote:
>>> +Bottlenecks to address:
>>> +
>>> +* frequent cancellation checks (intense OS interaction)
>> I don't understand this part: since when do cancellation checks involve
>> interaction with the OS? I thought the cancellation flag was an (atomic,
>> volatile) thing that requires neither cpu-level atomic operations, nor
>> higher-level locking.
>>
> It may not actually be "the OS" but the phenomenon is real.
> check_for_error() in ra_svn/editorp.c calls apr_poll(), which in
> turn calls select().
>
> To make things worse, we buffer small output data like props
> etc. but check for errors after each of them. So, more select()
> calls then socket writes ...
OK, so this is actually the server looking for abort commands from the
client, which is a bit different than the client checking the
cancellation flag. :)
> My idea is to short-circuit the check until at least x amount
> of data has been sent.
I'd expect that the same select/poll that waits for the write end of the
socket to become ready can also notice that the read end has data
available; making the whole thing asynchronous. Blindly sending data
after the client aborted the edit seems like the wrong way to solve
throughput issues.
Of course, taking a quick look at the implementation, this might involve
rewriting all of ra_svn's network layer ...
-- Brane
--
Certified & Supported Apache Subversion Downloads:
http://www.wandisco.com/subversion/download
Re: svn commit: r1387981 - /subversion/branches/10Gb/BRANCH-README
Posted by Stefan Fuhrmann <st...@wandisco.com>.
On Thu, Sep 20, 2012 at 9:22 PM, Branko Čibej <br...@wandisco.com> wrote:
> On 20.09.2012 13:41, stefan2@apache.org wrote:
> > +Bottlenecks to address:
> > +
> > +* frequent cancellation checks (intense OS interaction)
>
> I don't understand this part: since when do cancellation checks involve
> interaction with the OS? I thought the cancellation flag was an (atomic,
> volatile) thing that requires neither cpu-level atomic operations, nor
> higher-level locking.
>
It may not actually be "the OS" but the phenomenon is real.
check_for_error() in ra_svn/editorp.c calls apr_poll(), which in
turn calls select().
To make things worse, we buffer small output data like props
etc. but check for errors after each of them. So, more select()
calls then socket writes ...
My idea is to short-circuit the check until at least x amount
of data has been sent.
-- Stefan^2.
--
*
Join us this October at Subversion Live
2012<http://www.wandisco.com/svn-live-2012>
for two days of best practice SVN training, networking, live demos,
committer meet and greet, and more! Space is limited, so get signed up
today<http://www.wandisco.com/svn-live-2012>
!
*
Re: svn commit: r1387981 - /subversion/branches/10Gb/BRANCH-README
Posted by Branko Čibej <br...@wandisco.com>.
On 20.09.2012 13:41, stefan2@apache.org wrote:
> +Bottlenecks to address:
> +
> +* frequent cancellation checks (intense OS interaction)
I don't understand this part: since when do cancellation checks involve
interaction with the OS? I thought the cancellation flag was an (atomic,
volatile) thing that requires neither cpu-level atomic operations, nor
higher-level locking.
-- Brane
--
Certified & Supported Apache Subversion Downloads:
http://www.wandisco.com/subversion/download