You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Geoff Rowell <ge...@varolii.com> on 2009/06/05 18:40:49 UTC
Poor performance checking out working copy with many files (Windows
SVN 1.6.2)
I thought I'd share some difficulties that I encountered when getting a
working copy from a repository with a large number of files.
Background
--------------
The repository is running on a Windows 2003 unit. It's using CollabNet
Subversion Server 1.5.6 (Apache 2.2). It has a small number of revisions
(~130).
The repository contains a large number of small audio recordings. In
several cases there are over 5000 audio files in a folder.
My laptop is a Windows Vista system using CollabNet Subversion Client
1.6.2.
Working copy details:
- Size = 1.86 GB
- Size on disk = 3.19 GB
- # files = 480,000
- # folders = 860
Problem
----------
I needed to get a working copy on my laptop in order to add/update some
audio files. Since I knew this would be a performance challenge, I
suspended the TSVNCache process and attempted the checkout from the
command line. While it initially flew right along, eventually it slowed
down to the point where every file was taking about 30 seconds. I could
tell from the reported pathnames that the checkout was going to take
days (weeks?).
I decided to do some experiments to refine where the problem was.
I have an Ubuntu Linux box running Subversion 1.5.5. I used that to get
a working copy. It flew along at a steady rate and finished in 40
minutes. I archived the Linux working copy, transferred it to my laptop
and unpacked it. If the Windows file system had been a bottleneck, the
unpack would have shown a performance problem. It didn't. It unpacked it
at a steady rate in a couple of hours.
I wanted to determine if the problem would occur when using a Subversion
1.5 client under Windows, but I immediately ran into some problem masked
by issue #3102
<http://subversion.tigris.org/issues/show_bug.cgi?id=3102> . That
prevented me from investigating this further.
Luckily, I have the perfectly usable working copy, transferred from my
Linux box, that can be used for modifications. My typical commits of a
few files at a time don't suffer performance problems.
Thought I'd share this experience in hopes that something could be done
to improve the performance.
---
Geoff Rowell | SET Architect & SCC Administrator
Varolii Corporation | 781.482.2125 | FAX 781.482.2199 |
geoff.rowell@varolii.com <ma...@varolii.com>
This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender.
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2359797
To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].
RE: Poor performance checking out working copy with many files
(Windows SVN 1.6.2)
Posted by "Bolstridge, Andrew" <an...@intergraph.com>.
> -----Original Message-----
> From: Geoff Rowell [mailto:geoff.rowell@varolii.com]
> Sent: Friday, June 05, 2009 8:07 PM
> To: Andy Levy
> Cc: users@subversion.tigris.org
> Subject: RE: Poor performance checking out working copy with many
files
> (Windows SVN 1.6.2)
>
[snip]
> > There probably isn't much that can be done. NTFS and Windows do not
> > handle large numbers of files/directories in a single directory very
> > well, period. It's not so much a Subversion/library issue as it is
> > poor filesystem performance on that scale.
> >
> > Did you disable your on-access virus scanner while doing this
> > checkout? That's usually a real killer.
> >
>
> That was my first thought, but when the unpack (7-Zip) of the working
> copy
> archive didn't experience this problem I considered it to be unlikely.
>
I ran into this last week, so I checked - on Windows, the System process
was running at 100%, that turned out to be the virus checker getting
bogged down trying to scan. Disabling AV made the whole thing fly along
like it did when it started. I think this has something to do with the
number of temporary files that commit generates, I understand they're
looking into this for a fix along with the memory leak problem. See my
email from last week.
Andy
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2360240
To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].
Re: Poor performance checking out working copy with many files
(Windows SVN 1.6.2)
Posted by Parashkev Penev <pa...@mimoon.de>.
Hello,
There is already a ticket filed for your trouble together with a proof-
of-concept patch. I do no remember the exact number, but search for
"subversion file locking" and you should find it easily.
Br
Paro
On 05.06.2009, at 21:07, Geoff Rowell wrote:
> On Fri, Jun 5, 2009 at 15:02, Andy Levy <an...@gmail.com> wrote:
>> On Fri, Jun 5, 2009 at 14:40, Geoff Rowell <ge...@varolii.com>
> wrote:
>>> I thought I'd share some difficulties that I encountered when
> getting a
>>> working copy from a repository with a large number of files.
>>>
>>>
>>>
>>> Background
>>>
>>> --------------
>>>
>>> The repository is running on a Windows 2003 unit. It's using
> CollabNet
>>> Subversion Server 1.5.6 (Apache 2.2). It has a small number of
> revisions
>>> (~130).
>>>
>>> The repository contains a large number of small audio recordings. In
> several
>>> cases there are over 5000 audio files in a folder.
>>>
>>> My laptop is a Windows Vista system using CollabNet Subversion
> Client 1.6.2.
>>>
>>>
>>>
>>> Working copy details:
>>>
>>> - Size = 1.86 GB
>>>
>>> - Size on disk = 3.19 GB
>>>
>>> - # files = 480,000
>>>
>>> - # folders = 860
>>>
>>>
>>>
>>> Problem
>>>
>>> ----------
>>>
>>> I needed to get a working copy on my laptop in order to add/update
> some
>>> audio files. Since I knew this would be a performance challenge, I
> suspended
>>> the TSVNCache process and attempted the checkout from the command
> line.
>>> While it initially flew right along, eventually it slowed down to
> the point
>>> where every file was taking about 30 seconds. I could tell from the
> reported
>>> pathnames that the checkout was going to take days (weeks?).
>>>
>>>
>>>
>>> I decided to do some experiments to refine where the problem was.
>>>
>>>
>>>
>>> I have an Ubuntu Linux box running Subversion 1.5.5. I used that to
> get a
>>> working copy. It flew along at a steady rate and finished in 40
> minutes. I
>>> archived the Linux working copy, transferred it to my laptop and
> unpacked
>>> it. If the Windows file system had been a bottleneck, the unpack
> would have
>>> shown a performance problem. It didn't. It unpacked it at a steady
> rate in a
>>> couple of hours.
>>>
>>>
>>>
>>> I wanted to determine if the problem would occur when using a
> Subversion 1.5
>>> client under Windows, but I immediately ran into some problem masked
> by
>>> issue #3102. That prevented me from investigating this further.
>>>
>>>
>>>
>>> Luckily, I have the perfectly usable working copy, transferred from
> my Linux
>>> box, that can be used for modifications. My typical commits of a few
> files
>>> at a time don't suffer performance problems.
>>>
>>>
>>>
>>> Thought I'd share this experience in hopes that something could be
> done to
>>> improve the performance.
>>
>> There probably isn't much that can be done. NTFS and Windows do not
>> handle large numbers of files/directories in a single directory very
>> well, period. It's not so much a Subversion/library issue as it is
>> poor filesystem performance on that scale.
>>
>> Did you disable your on-access virus scanner while doing this
>> checkout? That's usually a real killer.
>>
>
> That was my first thought, but when the unpack (7-Zip) of the working
> copy
> archive didn't experience this problem I considered it to be unlikely.
>
> ---
> Geoff Rowell | SET Architect & SCC Administrator
> Varolii Corporation | 781.482.2125 | FAX 781.482.2199 |
> geoff.rowell@varolii.com
>
>
>
> This email and any files transmitted with it are confidential and
> intended solely for the use of the individual or entity to whom they
> are addressed. If you have received this email in error please
> notify the sender.
>
> ------------------------------------------------------
> http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2359808
>
> To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org
> ].
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2360212
To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].
RE: Poor performance checking out working copy with many files
(Windows SVN 1.6.2)
Posted by Geoff Rowell <ge...@varolii.com>.
On Fri, Jun 5, 2009 at 15:02, Andy Levy <an...@gmail.com> wrote:
> On Fri, Jun 5, 2009 at 14:40, Geoff Rowell <ge...@varolii.com>
wrote:
> > I thought I'd share some difficulties that I encountered when
getting a
> > working copy from a repository with a large number of files.
> >
> >
> >
> > Background
> >
> > --------------
> >
> > The repository is running on a Windows 2003 unit. It's using
CollabNet
> > Subversion Server 1.5.6 (Apache 2.2). It has a small number of
revisions
> > (~130).
> >
> > The repository contains a large number of small audio recordings. In
several
> > cases there are over 5000 audio files in a folder.
> >
> > My laptop is a Windows Vista system using CollabNet Subversion
Client 1.6.2.
> >
> >
> >
> > Working copy details:
> >
> > - Size = 1.86 GB
> >
> > - Size on disk = 3.19 GB
> >
> > - # files = 480,000
> >
> > - # folders = 860
> >
> >
> >
> > Problem
> >
> > ----------
> >
> > I needed to get a working copy on my laptop in order to add/update
some
> > audio files. Since I knew this would be a performance challenge, I
suspended
> > the TSVNCache process and attempted the checkout from the command
line.
> > While it initially flew right along, eventually it slowed down to
the point
> > where every file was taking about 30 seconds. I could tell from the
reported
> > pathnames that the checkout was going to take days (weeks?).
> >
> >
> >
> > I decided to do some experiments to refine where the problem was.
> >
> >
> >
> > I have an Ubuntu Linux box running Subversion 1.5.5. I used that to
get a
> > working copy. It flew along at a steady rate and finished in 40
minutes. I
> > archived the Linux working copy, transferred it to my laptop and
unpacked
> > it. If the Windows file system had been a bottleneck, the unpack
would have
> > shown a performance problem. It didn't. It unpacked it at a steady
rate in a
> > couple of hours.
> >
> >
> >
> > I wanted to determine if the problem would occur when using a
Subversion 1.5
> > client under Windows, but I immediately ran into some problem masked
by
> > issue #3102. That prevented me from investigating this further.
> >
> >
> >
> > Luckily, I have the perfectly usable working copy, transferred from
my Linux
> > box, that can be used for modifications. My typical commits of a few
files
> > at a time don't suffer performance problems.
> >
> >
> >
> > Thought I'd share this experience in hopes that something could be
done to
> > improve the performance.
>
> There probably isn't much that can be done. NTFS and Windows do not
> handle large numbers of files/directories in a single directory very
> well, period. It's not so much a Subversion/library issue as it is
> poor filesystem performance on that scale.
>
> Did you disable your on-access virus scanner while doing this
> checkout? That's usually a real killer.
>
That was my first thought, but when the unpack (7-Zip) of the working
copy
archive didn't experience this problem I considered it to be unlikely.
---
Geoff Rowell | SET Architect & SCC Administrator
Varolii Corporation | 781.482.2125 | FAX 781.482.2199 |
geoff.rowell@varolii.com
This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender.
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2359808
To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].
Re: Poor performance checking out working copy with many files
(Windows SVN 1.6.2)
Posted by Andy Levy <an...@gmail.com>.
On Fri, Jun 5, 2009 at 14:40, Geoff Rowell <ge...@varolii.com> wrote:
> I thought I’d share some difficulties that I encountered when getting a
> working copy from a repository with a large number of files.
>
>
>
> Background
>
> --------------
>
> The repository is running on a Windows 2003 unit. It’s using CollabNet
> Subversion Server 1.5.6 (Apache 2.2). It has a small number of revisions
> (~130).
>
> The repository contains a large number of small audio recordings. In several
> cases there are over 5000 audio files in a folder.
>
> My laptop is a Windows Vista system using CollabNet Subversion Client 1.6.2.
>
>
>
> Working copy details:
>
> - Size = 1.86 GB
>
> - Size on disk = 3.19 GB
>
> - # files = 480,000
>
> - # folders = 860
>
>
>
> Problem
>
> ----------
>
> I needed to get a working copy on my laptop in order to add/update some
> audio files. Since I knew this would be a performance challenge, I suspended
> the TSVNCache process and attempted the checkout from the command line.
> While it initially flew right along, eventually it slowed down to the point
> where every file was taking about 30 seconds. I could tell from the reported
> pathnames that the checkout was going to take days (weeks?).
>
>
>
> I decided to do some experiments to refine where the problem was.
>
>
>
> I have an Ubuntu Linux box running Subversion 1.5.5. I used that to get a
> working copy. It flew along at a steady rate and finished in 40 minutes. I
> archived the Linux working copy, transferred it to my laptop and unpacked
> it. If the Windows file system had been a bottleneck, the unpack would have
> shown a performance problem. It didn’t. It unpacked it at a steady rate in a
> couple of hours.
>
>
>
> I wanted to determine if the problem would occur when using a Subversion 1.5
> client under Windows, but I immediately ran into some problem masked by
> issue #3102. That prevented me from investigating this further.
>
>
>
> Luckily, I have the perfectly usable working copy, transferred from my Linux
> box, that can be used for modifications. My typical commits of a few files
> at a time don’t suffer performance problems.
>
>
>
> Thought I’d share this experience in hopes that something could be done to
> improve the performance.
There probably isn't much that can be done. NTFS and Windows do not
handle large numbers of files/directories in a single directory very
well, period. It's not so much a Subversion/library issue as it is
poor filesystem performance on that scale.
Did you disable your on-access virus scanner while doing this
checkout? That's usually a real killer.
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=1065&dsMessageId=2359804
To unsubscribe from this discussion, e-mail: [users-unsubscribe@subversion.tigris.org].