Re: Subversion 1.9.0-dev FSFS performance tests

[Johan Corveleyn <jc...@gmail.com>]

On Mon, Jun 30, 2014 at 6:21 PM, Ivan Zhakov <iv...@visualsvn.com> wrote:
> On 30 June 2014 18:51, Stefan Fuhrmann <st...@wandisco.com> wrote:
>> On Mon, Jun 30, 2014 at 4:06 PM, Ivan Zhakov <iv...@visualsvn.com> wrote:
>>>
>>> On 19 June 2014 14:21, Ivan Zhakov <iv...@visualsvn.com> wrote:
>>> > Hi,
>>> >
>>> > I've performed several FSFS performace tests using latest Subversion
>>> > from trunk@r1602928.
>>> >
>>> I've re-ran my FSFS performance tests with trunk@r1605444 using latest
>>> fsfs7 performance fixes including combining indexes to revision files.
>>
>>
> [..]
>
>> Also, it seems that some of these tests are run from hot
>> caches - causing a lot of variation and making comparison
>> pointless. An extreme case:
>>
>> ptime 1.0 for Win32, Freeware - http://www.pc-tools.net/
>> Copyright(C) 2002, Jem Berkes <jb...@pc-tools.net>
>>
>> ===  "svn log http://localhost/svn/ruby-fsfs6-unpacked >nul" ===
>>
>> Execution time: 216.064 s
>> ...
>> Execution time: 13.268 s
>> ...
>> Execution time: 18.061 s
>>
> Yes, I use hot caches and already noted this in my report: "Every test
> was run 3 times and only two latest used"
>
> I don't see the reason to test on cold disk caches because I assume
> that caches in the real servers are somewhat hot. No matter how it's
> complex to compare the results on hot disk caches. For me, log
> addressing feature is definitely useless if it slower on hot disk
> caches.

Innocent bystander's opinion: I appreciate your critical look at the
performance of fsfs7, Ivan, but I disagree. I think in lots of
real-world cases cold cache performance is much more important than
hot cache performance.

Especially in a big / busy repository and for use cases such as "svn
log". I usually don't ask the same "svn log" three times in a row.
User A might request "svn log" of some file or subtree, user B then
asks for another subtree, and so on. It's the performance of that
first (and usually only) request that matters most to me.

Same for export: different users export different parts of the
repository all the time. Not the same part a couple of times in a row
(except perhaps right after some release / milestone / ...).

So I for one am more interested in cold-cache performance
improvements, even if they come at a (hopefully small) hot-cache
performance cost.

OTOH, I do wonder about these (hot-cache) performance regressions ...
maybe they can still be improved?

-- 
Johan

Re: Subversion 1.9.0-dev FSFS performance tests

[Ivan Zhakov <iv...@visualsvn.com>]

On 7 July 2014 20:44, Julian Foad <ju...@btopenworld.com> wrote:
> I should probably let Stefan answer this, but...
>
> C. Michael Pilato wrote:
>>> On 07.07.2014 17:07, C. Michael Pilato wrote:
>>>> On 07/07/2014 10:58 AM, Ivan Zhakov wrote:
>>>>> My technical opinion that FSFS7/log addressing is slower by design,
>>>>> because it's doing more (read index, then read data instead of just
>>>>> read data) and only caching makes them comparable on performance to
>>>>> FSFS6 repositories.
>
> Ivan, it sounds like you've missed the important part of the design.
> It's designed to do LESS work in total, not more, because the cost of using
> the index is outweighed by the savings that it enables. As I understand it,
> a large saving is gained by re-ordering the data on disk during packing;
> that's why packing is essentially a requirement.
It seems to be designed to do less work in total if there is enough cache at
expense of other users who don't have enough caches.

As I understand constructing final content for file in FSFS7 works like
this (Stefan2 may correct me if I wrong):
1. Open rev10 file
2. Seek to end
3. Read rev10 file trailer
4. Seek to l2p index position
5. Read page of data from l2p index, decode it from 7bit encoding
6. Store index page content in cache.
7. Lookup index to find absolute offset of text delta
8. Open rev9 file
9. Seek to end
10. Read rev10 file trailer
11. Seek to l2p index position
12. Read page of data from l2p index, decode it from 7bit encoding
13. Store index page content in cache.
14. Seek to delta offset in rev10 file
15. Seek to delta offset in rev9 file

Then read two files combining deltas to produce final content.

Steps [2-7] and [9-13] could be potentially avoid if there is enough cache, but
for other use cases it's just waste of resources.

While for FSFS6 repository looks like:
1. Open rev10 file
2. Seek to delta offset in rev10 file
3. Open rev9 file
4. Seek to delta offset in rev9 file

Then read two files combining deltas to produce final content.

Regarding re-ordering during packing: it sounds good in theory, but I
doubt this will work in real world scenarios. There several reasons:
1. There is no explicit continous read: the code still reads
   data in random order and assume that OS or "block read" prefetch
(FSFS7 feature)
   will deliver data faster since it's reading around the same area.
2. It doesn't help if text deltas spreads across different pack files
3. SANs, SSDs, hybrid-disk and virtual disk are very popular now and assumption
   that they have spinning disks characteristcs is wrong IMHO.

---
Ivan Zhakov

Re: Subversion 1.9.0-dev FSFS performance tests

[Mark Phippard <ma...@gmail.com>]

On Mon, Jul 7, 2014 at 12:44 PM, Julian Foad <ju...@btopenworld.com>
wrote:

>
> C-Mike, my understanding is that F7 comprehensively beats F6 speed, by
> large factors around x1.5 and more, in the kind of scenarios it's designed
> for. Caching is an assumed part of the design. I don't know how much it
> needs for what scenarios, or what you mean by 'non-trivial' or 'ginormous',
> but probably not as much as you might be led to think. I suggest you first
> look at Stefan's recent-ish test results [1] and other recent threads.
>
> The only concerns about speed are about scenarios where it doesn't match
> F6 speed. While many scenarios are faster, some are slower, especially with
> the currently-default small cache size. Ivan reported some such scenarios
> [2].
>
> My general observation is that wider testing is needed to draw firm
> conclusions.
>
>
My feeling, which could be wrong, is that caching is of little value to me.
 All of the sites I work with are using Apache.  Most of them have hundreds
if not thousands of repositories.  Typically all of them are being used
each day and there are thousands of users hitting the site.

I just cannot envision scenarios where caching could ever be that helpful.
 Or that I could have enough RAM to make the caching useful while still
servicing hundreds of simultaneous Apache connections.

On Linux, we also use prefork MPM as that is the only one that ever seemed
stable over long periods.  I am guessing that the worker MPM could be tuned
to work well but we'd run into various problems with RAM not being released
and/or incompatibility with other modules running in the server and always
went back to prefork.

FWIW, I am fine with the idea of making a repository format that can give
great performance when caching is enabled.  I tend to think it should not
be our default format unless it performs just as well as what it is
replacing when cache is not being used, or is small.

-- 
Thanks

Mark Phippard
http://markphip.blogspot.com/

Re: Subversion 1.9.0-dev FSFS performance tests

[Julian Foad <ju...@btopenworld.com>]

I should probably let Stefan answer this, but...

C. Michael Pilato wrote:
>> On 07.07.2014 17:07, C. Michael Pilato wrote:
>>> On 07/07/2014 10:58 AM, Ivan Zhakov wrote:
>>>> My technical opinion that FSFS7/log addressing is slower by design,
>>>> because it's doing more (read index, then read data instead of just
>>>> read data) and only caching makes them comparable on performance to
>>>> FSFS6 repositories.

Ivan, it sounds like you've missed the important part of the design. It's designed to do LESS work in total, not more, because the cost of using the index is outweighed by the savings that it enables. As I understand it, a large saving is gained by re-ordering the data on disk during packing; that's why packing is essentially a requirement.

>>> I'm coming into this kinda late and after two weeks of vacation, so
>>> please forgive me if I misunderstand the above, but is it true that
>>> FSFS7 requires some kind of non-trivial caching just to match FSFS6's
>>> performance?
>> 
>> Yup.
> 
> May I then presume that for folks who have many repositories being
> hosted from a single server, FSFS7 will necessarily bring either a CPU
> performance hit (insufficient cache) or a RAM requirement/consumption
> hit (sufficient, ginormous cache)?  Or is the cache configuration
> perhaps per-server rather than per-repository?

C-Mike, my understanding is that F7 comprehensively beats F6 speed, by large factors around x1.5 and more, in the kind of scenarios it's designed for. Caching is an assumed part of the design. I don't know how much it needs for what scenarios, or what you mean by 'non-trivial' or 'ginormous', but probably not as much as you might be led to think. I suggest you first look at Stefan's recent-ish test results [1] and other recent threads.

The only concerns about speed are about scenarios where it doesn't match F6 speed. While many scenarios are faster, some are slower, especially with the currently-default small cache size. Ivan reported some such scenarios [2].

My general observation is that wider testing is needed to draw firm conclusions.

- Julian


[1] "Re: Current FSFS performance [RAW]" <http://svn.haxx.se/dev/archive-2014-07/0004.shtml>

[2] "Subversion 1.9.0-dev FSFS performance tests" <http://svn.haxx.se/dev/archive-2014-06/0065.shtml>

Re: Subversion 1.9.0-dev FSFS performance tests

[Stefan Fuhrmann <st...@wandisco.com>]

On Mon, Jul 7, 2014 at 8:59 PM, Branko Čibej <br...@wandisco.com> wrote:
> On 07.07.2014 20:44, Stefan Fuhrmann wrote:
>> ... it makes SVN parse the
>> whole 64k block that the OS provides anyway ...
>
> [off-topic]
>
> See, this is the kind of statement that makes me uneasy. You make an
> assertion about "the OS" (or about "the CPU" or "the architecture") as
> if it were generally true for any OS, or CPU, or architecture; even when
> it's clear that this is simply not the case. And your implementation
> typically presupposes these generalizations.

Well, the feature is opt-in to begin with and the block size can be
configured per repository. So, there is nothing that relies 64kB being
a magic number with the guts of the OS. Having fewer fopen() and
read() operations should also always be more efficient.

Things become particularly efficient, however, if requesting larger blocks
is actually not an overhead at all - and that is the case with many device
drivers, RAID controller or OS read-ahead settings etc. (64..256kB).
It is just a little bit of extra benefit, I'd like to tap (it's also why we use
aligned reads now where feasible) but it's in no way a precondition for
the "block-read" feature to be efficient.

-- Stefan^2.

Re: Subversion 1.9.0-dev FSFS performance tests

[Branko Čibej <br...@wandisco.com>]

On 07.07.2014 20:44, Stefan Fuhrmann wrote:
> ... it makes SVN parse the
> whole 64k block that the OS provides anyway ...

[off-topic]

See, this is the kind of statement that makes me uneasy. You make an
assertion about "the OS" (or about "the CPU" or "the architecture") as
if it were generally true for any OS, or CPU, or architecture; even when
it's clear that this is simply not the case. And your implementation
typically presupposes these generalizations.

I'm not saying that your optimizations have not been proven in general;
but we really shouldn't design our code based on assumptions about the
hardware or "the OS", and certainly shouldn't use them as arguments in
this particular discussion.

-- Brane

-- 
Branko Čibej | Director of Subversion
WANdisco | Realising the impossibilities of Big Data
e. brane@wandisco.com

Re: Subversion 1.9.0-dev FSFS performance tests

[Mark Phippard <ma...@gmail.com>]

On Mon, Jul 7, 2014 at 2:44 PM, Stefan Fuhrmann <
stefan.fuhrmann@wandisco.com> wrote:

> I just started the Windows tests in a "realistic" environment
> with a 4GB RAM server managing > 50GB of repository data.
> The goal is clearly that the default config is not slower than
> before and the computational overhead is roughly the same
> as we added when introducing manifest files for packed repos.
>

I have never considered the size of the repository in bytes that relevant
to performance, and other than the case of full text caches, I do not see
how it could matter.

For me, a realistic environment uses the Apache server and probably has at
least a few hundred repositories all of which are being used.  For our
larger users it would be thousands of active repositories.  Some of them
might be tens or hundreds of GB, but many of them are likely just a few
hundred MB.

I do not know what goes in the cache, but my assumption is that this sort
of environment would not lend itself well to caching of any kind.  We
currently use whatever is the default.

-- 
Thanks

Mark Phippard
http://markphip.blogspot.com/

Re: Subversion 1.9.0-dev FSFS performance tests

[Stefan Fuhrmann <st...@wandisco.com>]

On Mon, Jul 7, 2014 at 5:54 PM, C. Michael Pilato <cm...@collab.net> wrote:
> On 07/07/2014 11:23 AM, Branko Čibej wrote:
>> On 07.07.2014 17:07, C. Michael Pilato wrote:
>>> On 07/07/2014 10:58 AM, Ivan Zhakov wrote:
>>>> My technical opinion that FSFS7/log addressing is slower by design,
>>>> because it's doing more (read index, then read data instead of just
>>>> read data) and only caching makes them comparable on performance to
>>>> FSFS6 repositories.
>>> I'm coming into this kinda late and after two weeks of vacation, so
>>> please forgive me if I misunderstand the above, but is it true that
>>> FSFS7 requires some kind of non-trivial caching just to match FSFS6's
>>> performance?
>>
>> Yup.

Nope.

F7 is all about I/O reduction. No I/O, no reduction. The savings
are significant and a factor of 2 is typical. Even SSDs see speedups.

Data size and read operations (when to read what noderev / rep / ..)
are roughly unchanged. Thus, if caches are hot, the extra addressing
overhead cost you something between 0 (hot SVN caches) and
10% CPU (hot OS caches only).

F7 adds another feature that had to be made opt-in: "block-read".
Instead of reading only a few 100 bytes, it makes SVN parse the
whole 64k block that the OS provides anyway and puts the data
into cache. In environments with slow fopen(), that should save
CPU, but it requires significant SVN caches to be able to eventually
use the prefetched data. I'm particularly keen to see how much
of an impact that makes on Windows.

> May I then presume that for folks who have many repositories being
> hosted from a single server, FSFS7 will necessarily bring either a CPU
> performance hit (insufficient cache) or a RAM requirement/consumption
> hit (sufficient, ginormous cache)?  Or is the cache configuration
> perhaps per-server rather than per-repository?

I just started the Windows tests in a "realistic" environment
with a 4GB RAM server managing > 50GB of repository data.
The goal is clearly that the default config is not slower than
before and the computational overhead is roughly the same
as we added when introducing manifest files for packed repos.

The problem with all that measurement is that it is very hard
to create an environment that behaves roughly as the real
world would (multiple repositories being created over a long
period of time, interleaving each other). It took me a whole
day to rewrite the copy script that creates meaningful data
sets for systems whose operation cannot be controlled (SAN).

-- Stefan^2.

Re: Subversion 1.9.0-dev FSFS performance tests

["C. Michael Pilato" <cm...@collab.net>]

On 07/07/2014 11:23 AM, Branko Čibej wrote:
> On 07.07.2014 17:07, C. Michael Pilato wrote:
>> On 07/07/2014 10:58 AM, Ivan Zhakov wrote:
>>> My technical opinion that FSFS7/log addressing is slower by design,
>>> because it's doing more (read index, then read data instead of just
>>> read data) and only caching makes them comparable on performance to
>>> FSFS6 repositories.
>> I'm coming into this kinda late and after two weeks of vacation, so
>> please forgive me if I misunderstand the above, but is it true that
>> FSFS7 requires some kind of non-trivial caching just to match FSFS6's
>> performance?
> 
> Yup.

May I then presume that for folks who have many repositories being
hosted from a single server, FSFS7 will necessarily bring either a CPU
performance hit (insufficient cache) or a RAM requirement/consumption
hit (sufficient, ginormous cache)?  Or is the cache configuration
perhaps per-server rather than per-repository?

-- 
C. Michael Pilato <cm...@collab.net>
CollabNet   <>   www.collab.net   <>   Enterprise Cloud Development

Re: Subversion 1.9.0-dev FSFS performance tests

[Stefan Fuhrmann <st...@wandisco.com>]

On Tue, Jul 15, 2014 at 11:16 PM, Neels Hofmeyr <ne...@elego.de> wrote:
> On Mon, Jul 07, 2014 at 05:23:14PM +0200, Branko Čibej wrote:
>> On 07.07.2014 17:07, C. Michael Pilato wrote:
>> > On 07/07/2014 10:58 AM, Ivan Zhakov wrote:
>> >> My technical opinion that FSFS7/log addressing is slower by design,
>> >> because it's doing more (read index, then read data instead of just
>> >> read data) and only caching makes them comparable on performance to
>> >> FSFS6 repositories.
>> > I'm coming into this kinda late and after two weeks of vacation, so
>> > please forgive me if I misunderstand the above, but is it true that
>> > FSFS7 requires some kind of non-trivial caching just to match FSFS6's
>> > performance?
>>
>> Yup.
>
> <from the off>
> Sounds bad, but then again I remember that wc-ng's projected performance
> boost over 1.6 has not been evident from the start, either.
> "It's what you make of it" ??

Well, given that the detailed results for Windows are in now
as well, the key bits are

* from hot SVN caches, there is no difference between f6 and f7
* from hot OS caches, it's a mixed bag and depends on various factors
* reading from disk, f7 is faster for packed repos even with default configs
* "block-read" doubles the relative speedup but requires larger caches

-- Stefan^2.

Re: Subversion 1.9.0-dev FSFS performance tests

[Neels Hofmeyr <ne...@elego.de>]

On Mon, Jul 07, 2014 at 05:23:14PM +0200, Branko Čibej wrote:
> On 07.07.2014 17:07, C. Michael Pilato wrote:
> > On 07/07/2014 10:58 AM, Ivan Zhakov wrote:
> >> My technical opinion that FSFS7/log addressing is slower by design,
> >> because it's doing more (read index, then read data instead of just
> >> read data) and only caching makes them comparable on performance to
> >> FSFS6 repositories.
> > I'm coming into this kinda late and after two weeks of vacation, so
> > please forgive me if I misunderstand the above, but is it true that
> > FSFS7 requires some kind of non-trivial caching just to match FSFS6's
> > performance?
> 
> Yup.

<from the off>
Sounds bad, but then again I remember that wc-ng's projected performance
boost over 1.6 has not been evident from the start, either.
"It's what you make of it" ??

~Neels

Re: Subversion 1.9.0-dev FSFS performance tests

[Branko Čibej <br...@wandisco.com>]

On 07.07.2014 17:07, C. Michael Pilato wrote:
> On 07/07/2014 10:58 AM, Ivan Zhakov wrote:
>> My technical opinion that FSFS7/log addressing is slower by design,
>> because it's doing more (read index, then read data instead of just
>> read data) and only caching makes them comparable on performance to
>> FSFS6 repositories.
> I'm coming into this kinda late and after two weeks of vacation, so
> please forgive me if I misunderstand the above, but is it true that
> FSFS7 requires some kind of non-trivial caching just to match FSFS6's
> performance?

Yup.

-- 
Branko Čibej | Director of Subversion
WANdisco // Non-Stop Data
e. brane@wandisco.com

Re: Subversion 1.9.0-dev FSFS performance tests

["C. Michael Pilato" <cm...@collab.net>]

On 07/07/2014 10:58 AM, Ivan Zhakov wrote:
> My technical opinion that FSFS7/log addressing is slower by design,
> because it's doing more (read index, then read data instead of just
> read data) and only caching makes them comparable on performance to
> FSFS6 repositories.

I'm coming into this kinda late and after two weeks of vacation, so
please forgive me if I misunderstand the above, but is it true that
FSFS7 requires some kind of non-trivial caching just to match FSFS6's
performance?

-- 
C. Michael Pilato <cm...@collab.net>
CollabNet   <>   www.collab.net   <>   Enterprise Cloud Development

Re: Subversion 1.9.0-dev FSFS performance tests

[Ivan Zhakov <iv...@visualsvn.com>]

On 1 July 2014 03:27, Johan Corveleyn <jc...@gmail.com> wrote:
> On Mon, Jun 30, 2014 at 6:21 PM, Ivan Zhakov <iv...@visualsvn.com> wrote:
>> On 30 June 2014 18:51, Stefan Fuhrmann <st...@wandisco.com> wrote:
>>> On Mon, Jun 30, 2014 at 4:06 PM, Ivan Zhakov <iv...@visualsvn.com> wrote:
>>>>
>>>> On 19 June 2014 14:21, Ivan Zhakov <iv...@visualsvn.com> wrote:
>>>> > Hi,
>>>> >
>>>> > I've performed several FSFS performace tests using latest Subversion
>>>> > from trunk@r1602928.
>>>> >
>>>> I've re-ran my FSFS performance tests with trunk@r1605444 using latest
>>>> fsfs7 performance fixes including combining indexes to revision files.
>>>
>>>
>> [..]
>>
>>> Also, it seems that some of these tests are run from hot
>>> caches - causing a lot of variation and making comparison
>>> pointless. An extreme case:
>>>
>>> ptime 1.0 for Win32, Freeware - http://www.pc-tools.net/
>>> Copyright(C) 2002, Jem Berkes <jb...@pc-tools.net>
>>>
>>> ===  "svn log http://localhost/svn/ruby-fsfs6-unpacked >nul" ===
>>>
>>> Execution time: 216.064 s
>>> ...
>>> Execution time: 13.268 s
>>> ...
>>> Execution time: 18.061 s
>>>
>> Yes, I use hot caches and already noted this in my report: "Every test
>> was run 3 times and only two latest used"
>>
>> I don't see the reason to test on cold disk caches because I assume
>> that caches in the real servers are somewhat hot. No matter how it's
>> complex to compare the results on hot disk caches. For me, log
>> addressing feature is definitely useless if it slower on hot disk
>> caches.
>
> Innocent bystander's opinion: I appreciate your critical look at the
> performance of fsfs7, Ivan, but I disagree. I think in lots of
> real-world cases cold cache performance is much more important than
> hot cache performance.
>
> Especially in a big / busy repository and for use cases such as "svn
> log". I usually don't ask the same "svn log" three times in a row.
> User A might request "svn log" of some file or subtree, user B then
> asks for another subtree, and so on.

Caches on real-world server that I monitor are somewhat warm/hot for
busy repositories. The disk metadata be already cached, at least. But
I agree that querying "svn log" three times in a row for the same path
is too optimistic.

However, note that sequential updates for the same path is quite usual
for busy repositories.

> It's the performance of that first (and usually only) request that matters most to me.
>
I think we should care about the overall performance. On the other
hand, performance optimization should be targeted to the real
cases where users experience performance bootlenecks.

> Same for export: different users export different parts of the
> repository all the time. Not the same part a couple of times in a row
> (except perhaps right after some release / milestone / ...).
>
> So I for one am more interested in cold-cache performance
> improvements, even if they come at a (hopefully small) hot-cache
> performance cost.
>
> OTOH, I do wonder about these (hot-cache) performance regressions ...
> maybe they can still be improved?
>
My technical opinion that FSFS7/log addressing is slower by design,
because it's doing more (read index, then read data instead of just
read data) and only caching makes them comparable on performance to
FSFS6 repositories.

-- 
Ivan Zhakov