Performance tests - 1.7, serf and HTTPv2

[Mark Phippard <ma...@gmail.com>]

I ran the benchmark test against an http 1.6 and 1.7 server.  Details are here:

https://ctf.open.collab.net/sf/wiki/do/viewPage/projects.csvn/wiki/HTTPv2

The results are not encouraging.

* Compared to 1.6, checkout seems to be the biggest problem.

* HTTPv2 seems to help Neon more than it does Serf.  In many cases,
both are slower.

* Neon is generally still faster than Serf.  In some cases, the HTTPv2
changes seem to have widened the gap.

As I noted on the wiki, I am taking these numbers with a grain of
salt.  I think the tests just need to be run more times and by more
people before we draw conclusions.  I saw enough outliers during the
tests to say something was going on (perhaps Anti-Virus rearing its
head again?).

There is one issue that I want to raise that I do not think is an
outlier.  I was expecting HTTPv2 to yield significant improvements,
and so I stopped the Apache server after each test so I could grab its
logs.  I wanted to be able to show how much the HTTP traffic was
reduced.  I have not done this yet, but for Serf it looks like the
logs were bigger.  The issue I want to raise though is about Serf in
general.  Running these benchmarks with Neon yields an Access log of
about 102KB and a Subversion log of about 3KB.  Running the benchmarks
with Serf yields an Access log over 12MB and a Subversion log over 5
MB.  That concerns me.  I am just one user running only a handful of
commands against the server.  It looks like a real Subversion server
with just a few dozen active users will be generating logs in excess
of a GB every day.  At a minimum we are going to need to document this
in the release notes.  I can tell from users@ that most users do not
have log rotation set up.  I am concerned about how this might affect
performance of the server and it will certainly leave users vulnerable
to running out of disk space.

Other than looking at some of these logs to see if they show some kind
of problem in Serf, I do not know what else we can do.  We know Serf
generates many more requests than Neon when doing checkout/update.
Still, this is alarming when you see the reality of what it means.

-- 
Thanks

Mark Phippard
http://markphip.blogspot.com/

Re: Performance tests - 1.7, serf and HTTPv2

[Greg Stein <gs...@gmail.com>]

On Mon, Mar 28, 2011 at 18:14, Philip Martin <ph...@wandisco.com> wrote:
> Mark Phippard <ma...@gmail.com> writes:
>
>> There is one issue that I want to raise that I do not think is an
>> outlier.  I was expecting HTTPv2 to yield significant improvements,
>> and so I stopped the Apache server after each test so I could grab its
>> logs.  I wanted to be able to show how much the HTTP traffic was
>> reduced.  I have not done this yet, but for Serf it looks like the
>> logs were bigger.  The issue I want to raise though is about Serf in
>> general.  Running these benchmarks with Neon yields an Access log of
>> about 102KB and a Subversion log of about 3KB.  Running the benchmarks
>> with Serf yields an Access log over 12MB and a Subversion log over 5
>> MB.
>
> Yes, we discussed this earlier this month.  Serf makes more requests
> than neon for checkout/export/update, a GET and PROPFIND per file versus
> a single custom REQUEST.  My measurements using Subversion trunk as a
> dataset showed serf slightly faster than neon.  John's measurements with
> part of the Linux source tree showed serf slower than neon.  When I
> switched to his dataset I got the same result as he did.  The obvious
> difference between the datasets is that his was about half the size of
> mine but had about twice as many files.  So the serf approach of making
> lots of requests appears to be less efficient on lots of small files.

I'd be interested to see whether the checkout speed is client-bound or
server-bound. I'm assuming it is client, right?

The multiple request approach allows the server to provide for more
clients simultaneously. Lots of smaller requests rather than chugging
on one big response. For even larger provisioning, multiple servers
against one filesystem can service yet more clients simultaneously.

It also helps with proxying/caching, though Mark says SSL obviates
that (which I dispute, but it *is* valid to note that it is uncommon
and tricky to set up).

Lastly, the many-requests approach works better if the connection
drops for some reason and needs to be restarted. ra_serf can request
the bits that were dropped, while neon has to restart the whole
process (though I'm not even sure that neon has automatic
re-connection logic).

Cheers,
-g

Re: Performance tests - 1.7, serf and HTTPv2

[Philip Martin <ph...@wandisco.com>]

Mark Phippard <ma...@gmail.com> writes:

> There is one issue that I want to raise that I do not think is an
> outlier.  I was expecting HTTPv2 to yield significant improvements,
> and so I stopped the Apache server after each test so I could grab its
> logs.  I wanted to be able to show how much the HTTP traffic was
> reduced.  I have not done this yet, but for Serf it looks like the
> logs were bigger.  The issue I want to raise though is about Serf in
> general.  Running these benchmarks with Neon yields an Access log of
> about 102KB and a Subversion log of about 3KB.  Running the benchmarks
> with Serf yields an Access log over 12MB and a Subversion log over 5
> MB.

Yes, we discussed this earlier this month.  Serf makes more requests
than neon for checkout/export/update, a GET and PROPFIND per file versus
a single custom REQUEST.  My measurements using Subversion trunk as a
dataset showed serf slightly faster than neon.  John's measurements with
part of the Linux source tree showed serf slower than neon.  When I
switched to his dataset I got the same result as he did.  The obvious
difference between the datasets is that his was about half the size of
mine but had about twice as many files.  So the serf approach of making
lots of requests appears to be less efficient on lots of small files.

-- 
Philip