You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Xn Nooby <xn...@gmail.com> on 2007/05/07 18:55:19 UTC
Way to minimize number of svn files on a hardrive?
Is there a way to minimize the number of files that Subversion stores on a
hardrive? Our Windows XP PC's are apparently becoming somewhat unstable
after checking out about 100GB of small C sourcecode files, the number of
actual files is over a million. Our Windows Admin has asked me to look in
to this, in case there is a simple fix. I already disabled the TortoiseSVN
service, so that it only colorcodes directories being browsed (we use both
svn cli and gui). We are basically looking for a way to cut down the number
of files on the hardrive.
I had suggested moving the files to a network storage device or to a
directory on one of the servers, or possibly compressing (via NTFS) the
directories under Subversions control. I also considered doing an export
instead of a checkout, and then having people only checkout the directrories
they were working in (which is kind of kludgy).
Is there an option to maybe checkout just the working copies, and then
request the base copy in case you want to work on the file?
We have several large projects, and a few people work on all of them - they
are the ones having issues with too many files on their PC's. Sorry to be
vague about "their Windows problems", I have not seen them first hand. I
have not heard of any of the linux users having issues with too many files
on their drives.
Re: Way to minimize number of svn files on a hardrive?
Posted by Matt Sickler <cr...@gmail.com>.
try disabling window indexing service for at least the working copies
On 5/7/07, Xn Nooby <xn...@gmail.com> wrote:
>
>
> Is there a way to minimize the number of files that Subversion stores on a
> hardrive? Our Windows XP PC's are apparently becoming somewhat unstable
> after checking out about 100GB of small C sourcecode files, the number of
> actual files is over a million. Our Windows Admin has asked me to look in
> to this, in case there is a simple fix. I already disabled the TortoiseSVN
> service, so that it only colorcodes directories being browsed (we use both
> svn cli and gui). We are basically looking for a way to cut down the number
> of files on the hardrive.
>
> I had suggested moving the files to a network storage device or to a
> directory on one of the servers, or possibly compressing (via NTFS) the
> directories under Subversions control. I also considered doing an export
> instead of a checkout, and then having people only checkout the directrories
> they were working in (which is kind of kludgy).
>
> Is there an option to maybe checkout just the working copies, and then
> request the base copy in case you want to work on the file?
>
> We have several large projects, and a few people work on all of them -
> they are the ones having issues with too many files on their PC's. Sorry to
> be vague about "their Windows problems", I have not seen them first hand. I
> have not heard of any of the linux users having issues with too many files
> on their drives.
>
>
Re: Way to minimize number of svn files on a hardrive?
Posted by Talden <ta...@gmail.com>.
The issue would be the total number of file-system files and, to a
perhaps greater degree, their disposition.
Regardless, this would reduce the size of the problem though perhaps
not enough to solve the problem entirely.
I wonder whether it would be useful to have a property that indicated
a folder is 'write-once' such as tags, vendor drops etc... this could
indicate not to keep text bases or scan for local modifications.
checkout and update would want an option to override this so that the
content is modifiable for the less common situations.
This could significantly reduce the working copy sizes (pristine
copies at least as well as book-keeping perhaps) and perhaps even lend
itself to useful optimisations when (no I don't really believe it's an
if) the working copy format is changed...
--
Talden
On 5/8/07, Blair Zajac <bl...@orcaware.com> wrote:
>
> On May 7, 2007, at 9:15 PM, Ryan Schmidt wrote:
>
> >
> > On May 7, 2007, at 23:02, Blair Zajac wrote:
> >
> >>> You may be interested in SVK, which is based on Subversion and I
> >>> believe works with normal Subversion repositories. It stores its
> >>> local working copy data in a different format. Whether this takes
> >>> less or more disk space or uses fewer or more files than a
> >>> Subversion working copy I don't know. You could try it out.
> >>
> >> There is no .svn directory in an svk working copy, so you only
> >> have the files and directories that are checked out, so it is much
> >> smaller.
> >
> > Right, but don't you have to have a local mirror repository? How
> > much space does that take up?
>
> Well, it'll be as large as the master Subversion repository, or the
> subtree portion that you mirror. You can mirror subtrees of a
> repository.
>
> If you have multiple working copies from the same mirror, then it'll
> be less, presumably.
>
> I thought the issue was the number of files in the working copy, so
> this will help here.
>
> Regards,
> Blair
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: users-help@subversion.tigris.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Way to minimize number of svn files on a hardrive?
Posted by Blair Zajac <bl...@orcaware.com>.
On May 7, 2007, at 9:15 PM, Ryan Schmidt wrote:
>
> On May 7, 2007, at 23:02, Blair Zajac wrote:
>
>>> You may be interested in SVK, which is based on Subversion and I
>>> believe works with normal Subversion repositories. It stores its
>>> local working copy data in a different format. Whether this takes
>>> less or more disk space or uses fewer or more files than a
>>> Subversion working copy I don't know. You could try it out.
>>
>> There is no .svn directory in an svk working copy, so you only
>> have the files and directories that are checked out, so it is much
>> smaller.
>
> Right, but don't you have to have a local mirror repository? How
> much space does that take up?
Well, it'll be as large as the master Subversion repository, or the
subtree portion that you mirror. You can mirror subtrees of a
repository.
If you have multiple working copies from the same mirror, then it'll
be less, presumably.
I thought the issue was the number of files in the working copy, so
this will help here.
Regards,
Blair
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Way to minimize number of svn files on a hardrive?
Posted by Ryan Schmidt <su...@ryandesign.com>.
On May 7, 2007, at 23:02, Blair Zajac wrote:
>> You may be interested in SVK, which is based on Subversion and I
>> believe works with normal Subversion repositories. It stores its
>> local working copy data in a different format. Whether this takes
>> less or more disk space or uses fewer or more files than a
>> Subversion working copy I don't know. You could try it out.
>
> There is no .svn directory in an svk working copy, so you only have
> the files and directories that are checked out, so it is much smaller.
Right, but don't you have to have a local mirror repository? How much
space does that take up?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Way to minimize number of svn files on a hardrive?
Posted by Blair Zajac <bl...@orcaware.com>.
On May 7, 2007, at 6:29 PM, Ryan Schmidt wrote:
> On May 7, 2007, at 13:55, Xn Nooby wrote:
>
>> Is there a way to minimize the number of files that Subversion
>> stores on a hardrive? Our Windows XP PC's are apparently becoming
>> somewhat unstable after checking out about 100GB of small C
>> sourcecode files, the number of actual files is over a million.
>> Our Windows Admin has asked me to look in to this, in case there
>> is a simple fix. I already disabled the TortoiseSVN service, so
>> that it only colorcodes directories being browsed (we use both svn
>> cli and gui). We are basically looking for a way to cut down the
>> number of files on the hardrive.
>>
>> I had suggested moving the files to a network storage device or to
>> a directory on one of the servers, or possibly compressing (via
>> NTFS) the directories under Subversions control. I also considered
>> doing an export instead of a checkout, and then having people only
>> checkout the directrories they were working in (which is kind of
>> kludgy).
>>
>> Is there an option to maybe checkout just the working copies, and
>> then request the base copy in case you want to work on the file?
>>
>> We have several large projects, and a few people work on all of
>> them - they are the ones having issues with too many files on
>> their PC's. Sorry to be vague about "their Windows problems", I
>> have not seen them first hand. I have not heard of any of the
>> linux users having issues with too many files on their drives.
>
> There is no way to reduce the number of files in a Subversion
> working copy, other than checking out a smaller subset of the
> repository. The following feature requests may be relevant to you:
>
> Request to allow working copy with compressed text-base:
> http://subversion.tigris.org/issues/show_bug.cgi?id=908
>
> Request to allow working copy with no text-base:
> http://subversion.tigris.org/issues/show_bug.cgi?id=525
>
> It is possible to put your working copy on a remote server. At the
> web design company where I used to work, we did this, accessing our
> working copies from Windows machines via Samba shares stored on a
> Linux server. This works, but is considerably slower than having
> local working copies, and sometimes the TortoiseSVN badges get out
> of sync. In our case, we only did it this way so that we could test
> our web sites using the Apache, PHP and MySQL already installed on
> the Linux server, before committing changes from the working copy.
> If Windows is having trouble with your local working copies, my
> intuition would be to expect additional problems, not fewer
> problems, when moving the working copies to a server.
>
> You may be interested in SVK, which is based on Subversion and I
> believe works with normal Subversion repositories. It stores its
> local working copy data in a different format. Whether this takes
> less or more disk space or uses fewer or more files than a
> Subversion working copy I don't know. You could try it out.
There is no .svn directory in an svk working copy, so you only have
the files and directories that are checked out, so it is much smaller.
Regards,
Blair
--
Blair Zajac, Ph.D.
CTO, OrcaWare Technologies
<bl...@orcaware.com>
Subversion training, consulting and support
http://www.orcaware.com/svn/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Way to minimize number of svn files on a hardrive?
Posted by Ryan Schmidt <su...@ryandesign.com>.
On May 7, 2007, at 13:55, Xn Nooby wrote:
> Is there a way to minimize the number of files that Subversion
> stores on a hardrive? Our Windows XP PC's are apparently becoming
> somewhat unstable after checking out about 100GB of small C
> sourcecode files, the number of actual files is over a million.
> Our Windows Admin has asked me to look in to this, in case there is
> a simple fix. I already disabled the TortoiseSVN service, so that
> it only colorcodes directories being browsed (we use both svn cli
> and gui). We are basically looking for a way to cut down the
> number of files on the hardrive.
>
> I had suggested moving the files to a network storage device or to
> a directory on one of the servers, or possibly compressing (via
> NTFS) the directories under Subversions control. I also considered
> doing an export instead of a checkout, and then having people only
> checkout the directrories they were working in (which is kind of
> kludgy).
>
> Is there an option to maybe checkout just the working copies, and
> then request the base copy in case you want to work on the file?
>
> We have several large projects, and a few people work on all of
> them - they are the ones having issues with too many files on their
> PC's. Sorry to be vague about "their Windows problems", I have not
> seen them first hand. I have not heard of any of the linux users
> having issues with too many files on their drives.
There is no way to reduce the number of files in a Subversion working
copy, other than checking out a smaller subset of the repository. The
following feature requests may be relevant to you:
Request to allow working copy with compressed text-base:
http://subversion.tigris.org/issues/show_bug.cgi?id=908
Request to allow working copy with no text-base:
http://subversion.tigris.org/issues/show_bug.cgi?id=525
It is possible to put your working copy on a remote server. At the
web design company where I used to work, we did this, accessing our
working copies from Windows machines via Samba shares stored on a
Linux server. This works, but is considerably slower than having
local working copies, and sometimes the TortoiseSVN badges get out of
sync. In our case, we only did it this way so that we could test our
web sites using the Apache, PHP and MySQL already installed on the
Linux server, before committing changes from the working copy. If
Windows is having trouble with your local working copies, my
intuition would be to expect additional problems, not fewer problems,
when moving the working copies to a server.
You may be interested in SVK, which is based on Subversion and I
believe works with normal Subversion repositories. It stores its
local working copy data in a different format. Whether this takes
less or more disk space or uses fewer or more files than a Subversion
working copy I don't know. You could try it out.
http://svk.bestpractical.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Way to minimize number of svn files on a hardrive? - build system
Posted by MikeW <mw...@yahoo.co.uk>.
Xn Nooby <xnooby <at> gmail.com> writes:
>
> Thanks for all the suggestions! I will be trying them out, we are going to
first try moving the files to a server on the network, and let the users map a
remote drive to it. This is much slower, but the Windows workstations dont have
to manage all the files locally. One of our users is also having some success
by partitioning their work into several virtual machines (which is on a fast
host). Thanks again!
>
One build system (can't think of the name) uses a hierarchy whereby
your working files are stored on your local machine, other 'static'
files remain on a network server and are accessed there.
Anyone think of the name ?!
MikeW
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Way to minimize number of svn files on a hardrive?
Posted by Xn Nooby <xn...@gmail.com>.
Thanks for all the suggestions! I will be trying them out, we are going to
first try moving the files to a server on the network, and let the users map
a remote drive to it. This is much slower, but the Windows workstations dont
have to manage all the files locally. One of our users is also having some
success by partitioning their work into several virtual machines (which is
on a fast host). Thanks again!
Re: Way to minimize number of svn files on a hardrive?
Posted by Talden <ta...@gmail.com>.
If like us, you're stuck developing on windows then isn't much good
news for you. The NTFS filesystem performance does degrade as the
number of files/folders dramatically increases... It drags badly
enough with a few hundred thousand files... over a million has got to
hurt....
Partitioning the drive and spreading the working copies of the
projects across the partitions is an option. Unfortunately there
aren't many better options.
Windows really does need a filesystem refit... and then replace the
stuff around it.
--
Talden
On 5/8/07, Xn Nooby <xn...@gmail.com> wrote:
>
> Is there a way to minimize the number of files that Subversion stores on a
> hardrive? Our Windows XP PC's are apparently becoming somewhat unstable
> after checking out about 100GB of small C sourcecode files, the number of
> actual files is over a million. Our Windows Admin has asked me to look in
> to this, in case there is a simple fix. I already disabled the TortoiseSVN
> service, so that it only colorcodes directories being browsed (we use both
> svn cli and gui). We are basically looking for a way to cut down the number
> of files on the hardrive.
>
> I had suggested moving the files to a network storage device or to a
> directory on one of the servers, or possibly compressing (via NTFS) the
> directories under Subversions control. I also considered doing an export
> instead of a checkout, and then having people only checkout the directrories
> they were working in (which is kind of kludgy).
>
> Is there an option to maybe checkout just the working copies, and then
> request the base copy in case you want to work on the file?
>
> We have several large projects, and a few people work on all of them - they
> are the ones having issues with too many files on their PC's. Sorry to be
> vague about "their Windows problems", I have not seen them first hand. I
> have not heard of any of the linux users having issues with too many files
> on their drives.
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Way to minimize number of svn files on a hardrive? - compress;
links
Posted by Andreas Schweigstill <an...@schweigstill.de>.
Hello!
MikeW schrieb:
> (e.g. "Use link if working copy not modified")
That causes a hen and egg problem. WC file modifications are made by
other applications, e.g. text editor, and not by Subversion. When using
a link Subversion has no chance to compare what has been modified. Only
a timestamp and possibly a hash value of the last known file status
could be preserved (and stored in lots of small files...).
If the link directly points to the working copy file it will never be
detected as being modified because that is the concept of links. The
behaviour of hard links depends on how the file is accessed:
1. directly modify old file
2. rename old file, copy it to the original location, modify it
3. overwrite old file without keeping the original one
Regards
Andreas Schweigstill
--
Dipl.-Phys. Andreas Schweigstill
Schweigstill IT | Embedded Systems
Schauenburgerstraße 116, D-24118 Kiel, Germany
Phone: (+49) 431 5606-435, Fax: (+49) 431 5606-436
Mobile: (+49) 171 6921973, Web: http://www.schweigstill.de/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org