Current FSFS performance [RAW]

[Stefan Fuhrmann <st...@wandisco.com>]

I'm currently in travel limbo but I still want to share the
test results that I got with the latest f7 improvements.
They are attached in raw text format as produced by
the script and I will post a nicer version with proper
explanation later this week.

This is what I ran (all on the same machine):

  [f6 | f7]
x [ruby unpacked | ruby packed | bsd packed]
x [http | svn | file]
x [HDD HW raid | SSD SW raid | RAM disk]
x [slow default | medium | fast settings]

Basic results:

* f7 unpacked similar to f6 unpacked (+/-)
* packed significantly faster on HDD
* f7 packed ~2x as fast as packed f6 for export on HDD,
  details vary greatly with setup
* SSD or faster storage gets CPU bound with f7
  at 20 .. 60MB/s from cold disks, larger caches
  become important
* f7 commits are slower by usually ~5% with some variation

Settings (all uniform for all repos and storage devices):
* default - duh
* medium - 256MB cache
* fast - 1GB cache, revprop caching, bells & whistles

Numbers are the usual "elapsed, user, sys" in seconds.

-- Stefan^2.

Re: Current FSFS performance [RAW]

[Julian Foad <ju...@btopenworld.com>]

Thanks, Stefan.

BTW I really appreciate your taking the time to format the results into different comparison categories, with colour shades. Without that, given just a large table of raw numbers, I would not bother to spend enough time to interpret it. It is worth you, the writer, spending time on it so that we, the readers, don't have to.

- Julian



Stefan Fuhrmann wrote:
> On Wed, Jul 2, 2014 at 5:39 PM, Julian Foad wrote:
>> Only in the "SSD cold" column [...]
> 
> Well, I tried to express that with the "but cold caches" bit.
> Could probably use same punctuation there.
> 
>> I'd be interested to see a re-run of the HDD tests with this build 
>> (r1607306); do you plan to do that?
> 
> Can't until late September. However, I should get a Windows
> box soon to conduct testing in less controlled environment.
> 
>> For "load" what repository are you loading into (format? packed?) 
>> and if packed, how/when is it being packed?
> 
> Load is into f6 and f7, respectively as indicated in the tables
> (the dump files used are format independent). Repositories
> were left non-packed.
> 
>> Could you attach the actual scripts you use?
> 
> I simply hacked up a shell script and derived variations from
> it for the new runs. It requires superuser rights for the cache
> flushing bits and some paths have to be adjusted manually.
> Storage locations get symlinked to /files to unify access.
> 
> Scripts are provided as are without further commentary etc.
> 
>> In the later PDF file "load" is not mentioned in the summary of 
>> tests at the top of the file.
> 
> The load bit had been a late addition and I forgot to update the
> introduction page. Parameters are the same as for the first run.
> (It took 10h total to transform the raw logs into a spreadsheet -
> Formatting in LibreOffice turns out to be super buggy).
> 
>> Do you plan to re-test r1607306 over HTTP?
> 
> Not at the moment as the f6 / f7 relation seems to be the same
> for svn: as for http:. ra_serf results seem to be slightly less
> consistent / predictable than ra_svn, making it harder to tell
> whether a given test sequence got disturbed.
> 
> -- Stefan^2.

Re: Current FSFS performance [RAW]

[Stefan Fuhrmann <st...@wandisco.com>]

On Wed, Jul 2, 2014 at 5:39 PM, Julian Foad <ju...@btopenworld.com> wrote:
> Stefan Fuhrmann wrote:
>> Addendum. Julian found an omission in the last line of
>> explanations for "Export New" it should read:
>>
>> "_packed_ F7 consistently faster than F6 even with fast I/O but cold caches"
>
> Only in the "SSD cold" column; "SSD hot" f7/f6 ratio varies around +/- 5% with one outlier (BSD packed repo, "fast" system) where f7 is much slower than f6 (+31%).

Well, I tried to express that with the "but cold caches" bit.
Could probably use same punctuation there.

> I'd be interested to see a re-run of the HDD tests with this build (r1607306); do you plan to do that?

Can't until late September. However, I should get a Windows
box soon to conduct testing in less controlled environment.

> Also:
>
> For "load" what repository are you loading into (format? packed?) and if packed, how/when is it being packed?

Load is into f6 and f7, respectively as indicated in the tables
(the dump files used are format independent). Repositories
were left non-packed.

> Could you attach the actual scripts you use?

I simply hacked up a shell script and derived variations from
it for the new runs. It requires superuser rights for the cache
flushing bits and some paths have to be adjusted manually.
Storage locations get symlinked to /files to unify access.

Scripts are provided as are without further commentary etc.

> In the later PDF file "load" is not mentioned in the summary of tests at the top of the file.

The load bit had been a late addition and I forgot to update the
introduction page. Parameters are the same as for the first run.
(It took 10h total to transform the raw logs into a spreadsheet -
Formatting in LibreOffice turns out to be super buggy).

> Do you plan to re-test r1607306 over HTTP?

Not at the moment as the f6 / f7 relation seems to be the same
for svn: as for http:. ra_serf results seem to be slightly less
consistent / predictable than ra_svn, making it harder to tell
whether a given test sequence got disturbed.

-- Stefan^2.

Re: Current FSFS performance [RAW]

[Julian Foad <ju...@btopenworld.com>]

Stefan Fuhrmann wrote:
> Addendum. Julian found an omission in the last line of
> explanations for "Export New" it should read:
> 
> "_packed_ F7 consistently faster than F6 even with fast I/O but cold caches"

Only in the "SSD cold" column; "SSD hot" f7/f6 ratio varies around +/- 5% with one outlier (BSD packed repo, "fast" system) where f7 is much slower than f6 (+31%).

I'd be interested to see a re-run of the HDD tests with this build (r1607306); do you plan to do that?

Also:

For "load" what repository are you loading into (format? packed?) and if packed, how/when is it being packed?

Could you attach the actual scripts you use?

In the later PDF file "load" is not mentioned in the summary of tests at the top of the file.

Do you plan to re-test r1607306 over HTTP?

- Julian

Re: Current FSFS performance [RAW]

[Stefan Fuhrmann <st...@wandisco.com>]

Addendum. Julian found an omission in the last line of
explanations for "Export New" it should read:

"_packed_ F7 consistently faster than F6 even with fast I/O but cold caches"

Non-packed repos are not expected to perform better
in F7 than in F6, except in rare circumstances where
block-read may give an advantage even with single
revisions.

-- Stefan^2.

Re: Current FSFS performance [RAW]

[Stefan Fuhrmann <st...@wandisco.com>]

Here the processed results. They show the expected
benefits of packing as well as f7 on HDD. Issues found
with faster storage (such as hot disks) have been
addressed and the results measured and compiled as a
second PDF. The PDFs contain test setup descriptions,
compiled results and explanations for those results.

Major findings for Linux:
* There was significant CPU overhead for f7 with small
  caches. That's fixed now and access costs are roughly
  the same as in f6.
* Non-packed repos perform roughly the same in f6 and f7.
* Packed repos are faster in general and f7 is even faster,
  progressively so with larger caches - if there is I/O at all.

* Differences between RA layers are not format-specific.
* ra_serf delivers the least consistent performance as it
  seems to decide dynamically how many connections to
  use. A fast server answer seems, therefore, to lead to
  slower throughput *per client* (but higher in total). This
  is format independent.

Test config rationales:
* "slow" config is our very conservative default.
* "medium" is for users that have heard about caching
  and sprinkle "-M 256" like pixie dust everywhere.
* "fast" is for users that actually read documentation.
  They use larger caches (1G here to make it suitable
  for 32 bit systems as well) and enable all advanced
  options. All tests used the same options (no specific tuning).

Next steps - in roughly that order:
* I'll write up a wiki page on what makes repo testing
  hard and what I learned about it, i.e. how to conduct
  future performance testing.
* Review & comment on Ivan's results.
* Test Windows VM + SAN setup at our data center to
  get more data points. This will easily take a week to do.

-- Stefan^2.