Performance on large directories

[Stefan Fuhrmann <st...@alice-dsl.de>]

Hi there,

Here is my contribution to the performance awareness movement (PAM)
recently forming on this list. The attached script (bigdir.txt) will

* create a repository and start svnserve on it
   (to tell client and repository access performance apart)
* create folders with 1, 2, 4, 8, ... very small files in it
* perform add, st, ci, ls, up and co on them

The results showed an O(n^2) issue in c/i that I already fixed locally
and patches will be committed later this week. They simply need
some cleanup and proper commenting.

But they also show a significant regression in c/o performance on
the client-side. If run against a 1.7 server (because it supports turning
network data compression off), the difference is even larger.

So, there is strong evidence that WC-NG management code still
has significant processing and I/O overhead compared to 1.6.
The same seems to be the case for commit.

On a positive side, some operations are already much faster
in 1.7 than in 1.6. And I'm confident that the c/o and c/i overhead
can be fixed as well.

[1.6.x client, 1.6.x svnserve]
Processing 16384 files in the same folder
     Creating files ...      real   user    sys
     Adding files ...       119.818  116.959  2.624
     Running status ...     0.439  0.260  0.176
     Commit files ...       480.263  4.820  1.408
     Listing files ...      40.829  0.148  0.036
     Updating files ...     0.082  0.068  0.012
     Check out all ...      56.339  8.113  4.696

[1.6.x client, 1.7 HEAD svnserve]
Processing 16384 files in the same folder
     Creating files ...      real   user    sys
     Adding files ...       119.839  117.415  2.324
     Running status ...     0.419  0.272  0.148
     Commit files ...       486.165  4.940  1.484
     Listing files ...      1.111  0.132  0.032
     Updating files ...     0.072  0.048  0.016
     Check out all ...      8.932  5.660  3.264

[1.6.x client, 1.7 HEAD+patch svnserve]
Processing 16384 files in the same folder
     Creating files ...      real   user    sys
     Adding files ...       119.362  117.015  2.352
     Running status ...     0.428  0.292  0.136
     Commit files ...       16.357  4.944  1.476
     Listing files ...      1.102  0.128  0.028
     Updating files ...     0.066  0.056  0.012
     Check out all ...      9.054  5.672  3.256

[1.7 HEAD client, 1.7 HEAD svnserve]
Processing 16384 files in the same folder
     Creating files ...      real   user    sys
     Adding files ...       3.415  2.104  1.308
     Running status ...     4.525  3.592  0.916
     Commit files ...       511.538  24.846  8.661
     Listing files ...      1.038  0.144  0.040
     Updating files ...     1.091  1.044  0.016
     Check out all ...      24.929  15.753  9.173

[1.7 HEAD client, 1.7 HEAD+patch svnserve]
Processing 16384 files in the same folder
     Creating files ...      real   user    sys
     Adding files ...       3.489  2.108  1.364
     Running status ...     4.366  3.400  0.964
     Commit files ...       39.585  25.034  8.645
     Listing files ...      1.123  0.144  0.024
     Updating files ...     0.982  0.964  0.020
     Check out all ...      25.351  16.397  8.885

-- Stefan^2.