Data Point: File handling on Windows

[Stefan Fuhrmann <st...@alice-dsl.de>]

Hi there,

Last week on IRC I had a short discussion with Bert about
the file handling overhead on Windows. That prompted me
to run some tests on my notebook and this is what I found.

Opening or creating files and folders is not too expensive
even on NTFS. Creating and deleting folders is about as
expensive as creating or deleting empty files. Gains from
using multiple worker threads are measurable but scalability
is limited - at least as long as we operate on the same volume.
And even on Windows, c/o speeds >2000 files / sec are
technically feasible.

All tests are being executed repeatedly for 100 different
files in a single folder, no virus scanner active, 2 core
2.4GHz CPU; Windows XP 32 bits.

Results in detail shown as runs per second:

Test\#Threads    1    2    gain
(1)    5,639    6,798    21%
(2)    9,663    11,818    22%
(3)    3,943    5,292    34%
(4)    4,461    6,331    42%
(5)    4,460    5,842    31%
(6)    6,936    7,272    5%
(7)    8,305    9,661    16%

(1) ... create and close empty files
(2) ... delete these empty files
(3) ... create, write 12kB and close files
(4) ... rename these files
(5) ... delete these files
(6) ... create empty directories
(7) ... delete these empty directories

Test (1) with varying number of files per folder, 1 thread:
10    6,307
100    5,639
1000    5,083
10000    4,319

-- Stefan^2.

RE: Data Point: File handling on Windows

[Bert Huijben <be...@qqmail.nl>]

> -----Original Message-----
> From: Stefan Fuhrmann [mailto:stefanfuhrmann@alice-dsl.de]
> Sent: woensdag 27 april 2011 0:31
> To: Subversion Development
> Subject: Data Point: File handling on Windows
> 
> Hi there,
> 
> Last week on IRC I had a short discussion with Bert about
> the file handling overhead on Windows. That prompted me
> to run some tests on my notebook and this is what I found.

Before taking these numbers for 'normal results' can you verify what your
write caching settings are?

If your notebook tells Windows that it has a proper battery for your
harddisk windows reduces the number of full flushes for file operations,
because it knows that it won't see sudden power failures.

	Bert

Re: Data Point: File handling on Windows

[Ivan Zhakov <iv...@visualsvn.com>]

On Wed, Apr 27, 2011 at 02:30, Stefan Fuhrmann
<st...@alice-dsl.de> wrote:
> Hi there,
>
> Last week on IRC I had a short discussion with Bert about
> the file handling overhead on Windows. That prompted me
> to run some tests on my notebook and this is what I found.
>
> Opening or creating files and folders is not too expensive
> even on NTFS. Creating and deleting folders is about as
> expensive as creating or deleting empty files. Gains from
> using multiple worker threads are measurable but scalability
> is limited - at least as long as we operate on the same volume.
> And even on Windows, c/o speeds >2000 files / sec are
> technically feasible.
>

Btw, performance of creating files can be improved more using
FILE_ATTRIBUTE_TEMPORARY [1] flag in CreateFile function call.
Unfortunately APR doesn't provide such option.

[1] http://msdn.microsoft.com/en-us/library/aa363858%28v=vs.85%29.aspx

-- 
Ivan Zhakov