You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Joaquim Oliveira <jo...@atlantico.com.br> on 2007/02/06 19:40:38 UTC
Poor performance in windows. Switching back to CVS
Hi all,
Our team tried svn for 4 months, but we're switching back to CVS due to
performance problems. Our project is a huge java one: its structure has
10323 files and 2420 directories (117 MB). We made some tests using a
variety of tools:
- Eclipse 3.2 + Subclipse 1.1.9
- Eclipse 3.2 + Subversive 1.0.0 rc4
- Tortoise svn 1.4.1-7992
- Eclipse 3.2 + internal CVS support
- Tortoise CVS 1.8.30
SVN access was made using svn:// protocol. CVS was faster in all
situations. Most of then, it is twice faster. For example, an update in
working copy root folder was about 20% faster in CVS.
We noticed that SVN creates more administrative files and directories
than CVS. The checkout size is:
- CVS: 24849 files, 4841 folders. Disk usage: 164 MB
- SVN: 27450 files, 22319 folders (!). Disk usage: 261 MB
I searched the mail list archives, but couldn't find a solution for
this. I found something about "the NTFS file system does not perform
well when you have a large number of small files", but we need to
develop in Windows, so adopting Linux/Ext3 is not an option. I've
already seen these messages:
- http://svn.haxx.se/users/archive-2005-04/1557.shtml
-http://svn.haxx.se/users/archive-2005-04/1695.shtml
We already tried disabling anti-virus software and upgrading to the
latest version of the server (1.4.x) and plugins, but nothing worked.
Developers complain a lot about this and, although SVN features are
really better, a fast development environment is a must to our team.
Is there any way to improve SVN performance? What are the most common
bottlenecks?
Thanks in advance,
--
--
Joaquim Oliveira, MSc
Analista de Sistemas
Fone: +55 (85) 3216.7971
Fax: +55 (85) 3216.7864
Skype: joaquim.oliveira
ISO 9001 : 2000 - CMMI3
www.atlantico.com.br <http://www.atlantico.com.br>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Mike Brenner <mi...@mitre.org>.
>> Our team tried svn for 4 months, but we're switching back to CVS due
>> to performance problems. Our project is a huge java one: its structure
>> has 10323 files and 2420 directories (117 MB).
Similar experience here.
a. A windows java project with 21060 files in 16160 directories.
(SVN added about 80 percent of those directories and files.)
b. SVN takes about twice the time that cvs took for
both checkins and updates.
c. We often get the horrible "mismatches EOF"
messages, but after a few months of that we have
each picked a text editor that can normalize
the file for us before checking it in.
d. We have a lot of trouble cross-synchronizing
between svn servers.
e. We have a lot of trouble synchronizing individual
files that get taken home on thumb drives
to work on at home and then attempt to
un-Conflict them on Monday.
f. We have a lot of trouble tracking the last date
a file got modified (as opposed to the last date
it got checked in).
g. We will stay with svn because we need the ability to
svn copy and svn move things around.
h. Beware putting too many files in the same directory,
which kills Windows performance no matter what
configuration management software you use.
Mike Brenner
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by "B. Smith-Mannschott" <be...@gmail.com>.
On Feb 6, 2007, at 20:40, Joaquim Oliveira wrote:
> Hi all,
>
> Our team tried svn for 4 months, but we're switching back to CVS
> due to performance problems. Our project is a huge java one: its
> structure has 10323 files and 2420 directories (117 MB).
[snip]
This sounds similar to my situation. I work on a number of Java
projects, totalling some 13'000, all told. The largest single
project has roughly 9500 files in trunk.
>
> I searched the mail list archives, but couldn't find a solution for
> this. I found something about "the NTFS file system does not
> perform well when you have a large number of small files", but we
> need to develop in Windows, so adopting Linux/Ext3 is not an
> option. I've already seen these messages:
> - http://svn.haxx.se/users/archive-2005-04/1557.shtml
> -http://svn.haxx.se/users/archive-2005-04/1695.shtml
I've noticed this slow-down as well. My work machine is a 3 GHz P4
(2GB RAM) + a virus scanner (which seems to scan everything at the
slightest provocation) running WinXP. I have noticed the
sluggishness you describe. A simple update of a large working copy
and the hard drive rattles for minutes. The mind boggles.
Mostly, I've just avoided the problem by doing as much work as
possible on other machines:
I do most of my development on a 2.6GHz Linux box (1GB RAM) thanks to
the miracle of X11. It beats the XP box by 30% (clean and rebuild in
eclipse). It's far faster for file-system intensive operations. SVN
really does like to crawl all over the disk, doesn't it?
Recently I've been moving more of my development work onto my
personal notebook: A dual core MacBook Pro. I use Parallels Desktop
(virtualization) to run Windows when necessary. I'd move to the mac
full-time only there's a problem relating to composed and decomposed
combining characters which prevents me from working with file names
containing non-ascii characters.
Another recent discovery, that I find faster for some operations is
svk, but it (1) has no gui and (2) doesn't seem to follow svn
externals, of which we make fairly heavy use in our repository.
Sorry, bit of a ramble. You're right. Svn is slow(er) on Windows
(than I'd like).
My coping strategy has been to use alternatives when possible. In my
case this works well because my target is the JVM and my primary
tools either run on the JVM (eclipse, oXygen, rational modeller) or
are widely ported (EMACS, xsltproc, xmlstar, python, ...)
// ben
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Julian Hsiao <ma...@nyanisore.net>.
Joaquim Oliveira <jo...@atlantico.com.br> wrote:
> I made a test using Ext2IFS (http://www.fs-driver.org/). Although the
> filesystem is EXT3, the performance in Windows XP SP2 was worst than
> NTFS. Do you know other drives whose performance is better?
Try ext2fsd <ext2fsd.sourceforge.net> and see if you have better luck with
it. I've not used either drivers, so I can't tell you how it will perform.
Good Luck.
Julian Hsiao
madoka@nyanisore.net
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Joaquim Oliveira <jo...@atlantico.com.br>.
Julian Hsiao escreveu:
> There are kernel drivers for 2k/XP for ext2. If the bottleneck is indeed
> NTFS, perhaps you can create a ext2 partition specifically for your working
> copy.
>
> It's worth a shot, no?
>
> Julian Hsiao
> madoka@nyanisore.net
Julian,
I made a test using Ext2IFS (http://www.fs-driver.org/). Although the
filesystem is EXT3, the performance in Windows XP SP2 was worst than
NTFS. Do you know other drives whose performance is better?
Thanks for your help,
Joaquim Oliveira
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Phillip Susi <ps...@cfl.rr.com>.
Joaquim Oliveira wrote:
> Phillip,
>
> I think you didn't get the point: our server is a linux box. The
> overhead is in developer's machine (WinXP) due to the huge number of
> folders and files created by svn working copy.
Oh my... are you managing an active branch containing tens of thousands
of files then? SVN only keeps double the local files in a working copy
for fast diff and status. I would imagine that if you had enough files
to cause a serious problem only cutting them in half would not make that
much difference... certainly not enough to outweigh the delay cvs incurs
of having to contact the server for a status or diff.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Joaquim Oliveira <jo...@atlantico.com.br>.
Phillip Susi escreveu:
> Or you can use a db backend, which uses far fewer files, instead of
> the fs backend.
>
> And just because you develop on windows does not mean your server must
> run windows. I develop an embedded system on windows because that is
> where the vendor tools are supplied, but I migrated our server to
> linux to host the svn repo and bug tracking database, because these
> tools simply run better under linux and the server is more reliable too.
Phillip,
I think you didn't get the point: our server is a linux box. The
overhead is in developer's machine (WinXP) due to the huge number of
folders and files created by svn working copy.
Joaquim Oliveira
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Jan Hendrik <ja...@myrealbox.com>.
Concerning Re: Poor performance in windows. Sw
Ryan Schmidt wrote on 12 Feb 2007, 16:31, at least in part:
>
> On Feb 12, 2007, at 15:31, Jan Hendrik wrote:
>
> > OH, BTW there is a bug in SVN: if a file is modified without
> > changing timestamp and/or filesize, SVN will not commit the file,
> > and I think if the file is updated local changes are lost.
>
> This behavior does exist in Subversion, but I do not believe it is
> considered a bug. Rather, I believe it is considered a user error to
> modify a file but not update its timestamp.
Well, is it really to be considered a user error? There are more
than one apps out there that deliberately or by user's choice do not
touch the timestamp on search&replace operations for one or other
reason.
The case I came over this behaviour was different though: wanting
to upload the correction of a bad typo (filesize did not change)
immediately from a working copy normally not used for uploading I
had the FTP sync app setting local timestamps to server
timestamps so that files different would be marked and not
everything else would be uploaded jst because of the difference of
the timestamp, too - and voila: the typo corrections were not
committed and I had to redo the typo thing on another WC. The
user error in this case was not doing the upload through the usual
WC, but not the unchanged timestamp.
Besides, if the timestamp is reset to an older date than in the base
copy SVN will commit even with filesize unchanged though this
would even more indicate a user error.
(The live server is hosted outside and no working copy, but
synchronized through FTP, with local timestamps set to server
timestamps after upload; no SVN on the server, repos neither
accessible from outside.)
JH
---------------------------------------
Freedom quote:
The Founding Fathers knew
a government can't control the economy
without controlling people.
And they knew when a government sets out to do that,
it must use force and coercion to achieve its purpose.
So we have come to a time for choosing.
-- Ronald Reagan, October 27, 1964
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
RE: Poor performance in windows. Switching back to CVS
Posted by Jan Hendrik <ja...@myrealbox.com>.
Concerning RE: Poor performance in windows. Sw
Gavin Lambert wrote on 14 Feb 2007, 12:09, at least in part:
> Quoth Jan Hendrik <ma...@myrealbox.com>:
> >> This behavior does exist in Subversion, but I do not believe it is
> >> considered a bug. Rather, I believe it is considered a user error
> >> to modify a file but not update its timestamp.
> >
> > A timestamp reset to an earlier date should be considered even
> > more a user error then, but SVN will happily commit then. At least
> > SVN should provide a --force switch to have it compare ALL files in
> > WC with base, no matter what timestamps say.
>
> Actually no, it won't commit then. At least not if there aren't any
> actual changes.
That's right & OK.
> The timestamp thing is actually a *feature* of SVN, not a bug. It's a
> performance optimisation, since it's a lot easier and faster to check
> the timestamps of files than to check their contents.
True.
> Essentially, when scanning the directory SVN will first obtain a list
> of all files whose timestamps are different from what it was
> expecting. It then scans the *contents* of those files having a
> different timestamp to see if the actual file contents have changed
> (by doing a hash of some kind, probably an MD5, but I'm not sure of
> the exact details). Only those files which have both a different
> timestamp and a different checksum will be considered "modified" and
> will be committed.
>
> You can test this easily yourself -- from a pristine unmodified (but
> versioned) file, make a change and save it. Do "svn st" and note the
> file is listed as modified. Now edit the file again and remove the
> change you just made. Do another "svn st" and the file will no longer
> be listed (because it's no longer considered modified, even though its
> timestamp differs).
I know. Happens all the time here with the WC normally used for
uploading to the live server: commits before the upload are not
committed again, though the upload process changes timestamps.
> If it scanned the contents of all files then the update/commit process
> would be at least an order of magnitude slower than it already is.
That's where the --force switch would come in: Force SVN to do a
hash check, kind of make sure about modifications, no matter what
the first-level timestamp check says. Using a switch like this the
user usually knows what he does.
TortoiseSVN reportedly does kind of similar with their
implementation of cleanup, though obviously rather primitive:
correcting the timestamps in base copy so otherwise not modified
files are no longer scanned. As it did not help me in my case of
unchanged filesizes it looks like TSVN rests on filesize, not
checksums for this:
if ((timestampWC != timestampBC) &&(filesizeWC == filesize BC)):
timestampBC t= timestampWC
JH
---------------------------------------
Freedom quote:
Capitalism is not an 'ism.'
It is closer to being the opposite of an 'ism,'
because it is simply the freedom of ordinary people
to make whatever economic transactions they can mutually agree to.
-- Thomas Sowell
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
RE: Poor performance in windows. Switching back to CVS
Posted by Gavin Lambert <ga...@compacsort.com>.
Quoth Jan Hendrik <ma...@myrealbox.com>:
>> This behavior does exist in Subversion, but I do not believe it is
>> considered a bug. Rather, I believe it is considered a user error to
>> modify a file but not update its timestamp.
>
> A timestamp reset to an earlier date should be considered even
> more a user error then, but SVN will happily commit then. At least
> SVN should provide a --force switch to have it compare ALL files in
> WC with base, no matter what timestamps say.
Actually no, it won't commit then. At least not if there aren't any
actual changes.
The timestamp thing is actually a *feature* of SVN, not a bug. It's a
performance optimisation, since it's a lot easier and faster to check
the timestamps of files than to check their contents.
Essentially, when scanning the directory SVN will first obtain a list of
all files whose timestamps are different from what it was expecting. It
then scans the *contents* of those files having a different timestamp to
see if the actual file contents have changed (by doing a hash of some
kind, probably an MD5, but I'm not sure of the exact details). Only
those files which have both a different timestamp and a different
checksum will be considered "modified" and will be committed.
You can test this easily yourself -- from a pristine unmodified (but
versioned) file, make a change and save it. Do "svn st" and note the
file is listed as modified. Now edit the file again and remove the
change you just made. Do another "svn st" and the file will no longer
be listed (because it's no longer considered modified, even though its
timestamp differs).
If it scanned the contents of all files then the update/commit process
would be at least an order of magnitude slower than it already is.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Jan Hendrik <ja...@myrealbox.com>.
Concerning Re: Poor performance in windows. Sw
Ryan Schmidt wrote on 12 Feb 2007, 16:31, at least in part:
>
> On Feb 12, 2007, at 15:31, Jan Hendrik wrote:
>
> > OH, BTW there is a bug in SVN: if a file is modified without
> > changing timestamp and/or filesize, SVN will not commit the file,
> > and I think if the file is updated local changes are lost.
>
> This behavior does exist in Subversion, but I do not believe it is
> considered a bug. Rather, I believe it is considered a user error to
> modify a file but not update its timestamp.
A timestamp reset to an earlier date should be considered even
more a user error then, but SVN will happily commit then. At least
SVN should provide a --force switch to have it compare ALL files in
WC with base, no matter what timestamps say.
JH
---------------------------------------
Freedom quote:
All the great things are simple,
and many can be expressed in a single word:
freedom, justice, honor, duty, mercy, hope.
-- Sir Winston Churchill
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Sindbad the Seafarer <si...@gmx.net>.
Concerning Re: Poor performance in windows. Sw
Ryan Schmidt wrote on 12 Feb 2007, 16:31, at least in part:
>
> On Feb 12, 2007, at 15:31, Jan Hendrik wrote:
>
> > OH, BTW there is a bug in SVN: if a file is modified without
> > changing timestamp and/or filesize, SVN will not commit the file,
> > and I think if the file is updated local changes are lost.
>
> This behavior does exist in Subversion, but I do not believe it is
> considered a bug. Rather, I believe it is considered a user error to
> modify a file but not update its timestamp.
A timestamp reset to an earlier date should be considered even
more a user error then, but SVN will happily commit then. At least
SVN should provide a --force switch to have it compare ALL files in
WC with base, no matter what timestamps say.
JH
---------------------------------------
Freedom quote:
Nothing can alter my inner soul:
I shall pursue my own straight course
and shall do what I believe to be right and honorable.
-- Frederick the Great
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Jan Hendrik <ja...@myrealbox.com>.
Concerning Re: Poor performance in windows. Switching back t
Les Mikesell wrote on 16 Feb 2007, 8:02, at least in part:
> Jan Hendrik wrote:
>
> >>> Maybe there are tools that handle this
> >>> just the other way round, but WS-FTP did not nor does
> >>> TotalCommander. Probably one could automize this with a post-
> >>> commit hook and rsync, but this also takes a lot of control out of
> >>> it.
> >> So do an 'rsync -av' yourself.
> >
> > At first sight rsync sounds fascinating, I have had a look at it in
> > the past and now again, but it is not for us.
>
> Try it before making that decision. It's free.
Yep, just kind of confirmed (in a telnet session) that rsync is
available on all the domains here, hopefully also for local-remote
connections. Will have to take a free weekend to get acquainted
with this stuff.
> If you used Linux for this, migration cost would be 1 so-so PC to run
> it
> plus free software. Or if you have sufficient CPU/RAM on an existing
>
> box you could run it under the free VMware server. You could let it
> run your subversion repository too - and be a samba file server for
> your windows boxes. But, the cywin version of rsync works fine.
Naw, the only so-so PC here is absolutely broken - museum-
quality. ;) A Samba server is what the NAS Linkstation is, but it
doesn't work for the main chunk of to-be-shared data here as Word
prevents any PC from standby/hibernation as long as a file on a
share is open. That's why I tried OpenOffice last year, but
migrating the 10-15% of docs that need manual attention would
keep me busy for months, plus rewriting several macros. Too
expensive.
> > have checked this only last year for just Word97->OpenOffice) and
> > running an OS layer (cygwin) on top of Windows is not very
> > intrigueing either.
>
> Intrigueing? It's just a dll library that maps unix system calls into
> the windows equivalents so the apps compile unchanged. You don't need
> all the other tools if you don't want them, although you probably need
> ssh which rsync will run transparently for you.
Understood cygwin as some virtual machine or layer, not just a
DLL.
> GUI's are pretty and nice for doing things once. If you do them
> repeatedly it is even nicer to have the job fully scripted so you
> don't have to use any interface - and the scripting is easy with
> command line tools.
Sure, did some scripting myself over the years and especially
lately, just don't feel comfortable about synching the webserver w/o
seeing what happens (or only afterwards from the output directed
into a file). Also would have to make sure that files generated by
scripts on the webserver (and only there) won't be deleted, now and
in future - automation often lets forget what is happening and then
one wastes hours and days to understand why something works
locally and not by cron on the remote server.
Anyway, lots of thanks, Les!
JH
---------------------------------------
Freedom quote:
The price of freedom is eternal vigilance.
-- Thomas Jefferson
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Jan Hendrik <ja...@myrealbox.com>.
Concerning Re: Poor performance in windows. Sw
Jeff Smith wrote on 16 Feb 2007, 11:01, at least in part:
> On Friday 16 February 2007 10:01, Jan Hendrik wrote:
> > Naw, the only so-so PC here is absolutely broken - museum-
> > quality. ;) A Samba server is what the NAS Linkstation is, but it
> > doesn't work for the main chunk of to-be-shared data here as Word
> > prevents any PC from standby/hibernation as long as a file on a
> > share is open. That's why I tried OpenOffice last year, but
> > migrating the 10-15% of docs that need manual attention would keep
> > me busy for months, plus rewriting several macros. Too expensive.
>
> Sorry I'm still off subject a little, but it kinda puzzled me when you
> said, "We don't have Linux here (and will not for the forseeable
> future, migration cost is too high...", Something bothered me--from
> the beginning of this discussion, you have been revealing
> cost-after-cost related to *not* migrating... Hmmm. The bottom line
> for us was that if migrating to GNU/Linux costs more than staying with
> microsoft, then we were going about the migration the wrong way
> (asking for the trouble you're getting).
Sorry for having you puzzled and distracted, Jeff. The cost thing
has nothing to do with the original issue, it originates in an aside of
mine (I really like asides, don't I? Already this whole string of this
thread originates in an aside) for why I had reservations about
rsync since I did not realize that we neither would need a Linux box
for just this nor cygwin is no full-blown virtual machine I would not
consider at all. As Les then suggested further advantages of a
Linux box as fileserver etc. I added the above on what we had tried
(and failed) with a Linkstation as NAS fileserver recently.
We are a small shop here. We can't just say OK, the webpage
fellows can switch to Linux immediately, the sales department may
start by using OpenOffice and gradually convert Word stuff till they
set for changing to Linux, while the accounting must stick with
Windows as long as there is no accounting application approved by
the financial authority overhere, and in the meantime we set up a
big Linux server serving all. We would have to do such a change at
once, including a steep learning curve for Linux, and therefore must
see if the complete picture figures. For us fellows it's an either/or
or we would end up booting back and forth (and a dedicated 24h
server just overkill).
> I paid nothing but effort for my SUSE 10. You obviously have not saved
> effort by sticking with what you had. You are only saving yourself
> from any chance of resolving your performance problem. I guess that's
> all the more I could say...
As time allows I intend to set up Linux in a dual boot for getting
more acquainted with it, but this is an intention I carry on since
long... :(
JH
---------------------------------------
Freedom quote:
We who live in free market societies believe
that growth, prosperity and ultimately human fulfillment,
are created from the bottom up, not the government down.
Only when the human spirit is allowed to invent and create,
only when individuals are given a personal stake
in deciding economic policies and benefiting from their success --
only then can societies remain economically alive, dynamic,
progressive, and free. Trust the people.
This is the one irrefutable lesson of the entire postwar period
contradicting the notion that rigid government controls
are essential to economic development.
-- Ronald Reagan, September 29, 1981
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Jeff Smith <js...@robotronics.com>.
On Friday 16 February 2007 10:01, Jan Hendrik wrote:
> Naw, the only so-so PC here is absolutely broken - museum-
> quality. ;) A Samba server is what the NAS Linkstation is, but it
> doesn't work for the main chunk of to-be-shared data here as Word
> prevents any PC from standby/hibernation as long as a file on a
> share is open. That's why I tried OpenOffice last year, but
> migrating the 10-15% of docs that need manual attention would
> keep me busy for months, plus rewriting several macros. Too
> expensive.
Sorry I'm still off subject a little, but it kinda puzzled me when you
said, "We don't have Linux here (and will not for the forseeable
future, migration cost is too high...", Something bothered me--from
the beginning of this discussion, you have been revealing
cost-after-cost related to *not* migrating... Hmmm. The bottom line
for us was that if migrating to GNU/Linux costs more than staying
with microsoft, then we were going about the migration the wrong way
(asking for the trouble you're getting).
I paid nothing but effort for my SUSE 10. You obviously have not saved
effort by sticking with what you had. You are only saving yourself
from any chance of resolving your performance problem. I guess that's
all the more I could say...
Except if I understand your situation, I was wondering how long it
would take to `touch` each of the working files of the shared WC,
because that would be the quickest way to update every timestamp and
force subversion to compare each to see which have changed. You would
have to make sure `touch` doesn't get the copies within ".svn" of
course.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Les Mikesell <le...@gmail.com>.
Jan Hendrik wrote:
>>> Maybe there are tools that handle this
>>> just the other way round, but WS-FTP did not nor does
>>> TotalCommander. Probably one could automize this with a post-
>>> commit hook and rsync, but this also takes a lot of control out of
>>> it.
>> So do an 'rsync -av' yourself.
>
> At first sight rsync sounds fascinating, I have had a look at it in the
> past and now again, but it is not for us.
Try it before making that decision. It's free.
> We don't have Linux here
> (and will not for the forseeable future, migration cost is too high,
If you used Linux for this, migration cost would be 1 so-so PC to run it
plus free software. Or if you have sufficient CPU/RAM on an existing
box you could run it under the free VMware server. You could let it run
your subversion repository too - and be a samba file server for your
windows boxes. But, the cywin version of rsync works fine.
> have checked this only last year for just Word97->OpenOffice) and
> running an OS layer (cygwin) on top of Windows is not very
> intrigueing either.
Intrigueing? It's just a dll library that maps unix system calls into
the windows equivalents so the apps compile unchanged. You don't need
all the other tools if you don't want them, although you probably need
ssh which rsync will run transparently for you.
> Besides until recently rsync was not provided by
> the webhoster, and I think it is still not available on all of the
> domains here. And the way Total Commander handles FTP synch
> leaves a lot of control for me, more with far less trouble than the
> commandline tool could (I use the commandline for many things).
> It's a difference similar between committing with SVN commandline
> and TortoiseSVN where file selection/adding, commit message, diff
> are all in one dialog. The Windows rsync GUIs I just checked
> OTOH seem to be either remote backup systems just happening to
> use rsync protocol a/o requiring to be run on both systems as
> client/server - again out of my control for the webserver side.
GUI's are pretty and nice for doing things once. If you do them
repeatedly it is even nicer to have the job fully scripted so you don't
have to use any interface - and the scripting is easy with command line
tools.
--
Les Mikesell
lesmikesell@gmail.com
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Jan Hendrik <ja...@myrealbox.com>.
Concerning Re: Poor performance in windows. Switching back t
Les Mikesell wrote on 15 Feb 2007, 11:33, at least in part:
> Jan Hendrik wrote:
>
> > FTP synching e.g. Not everyone has his own live server standing in
> > the own basement and not every hoster provides SVN to make the live
> > copy just another WC. As long as _always_ done from the same WC
> > there's no issue,
>
> Of course there are issues. You have to do adds, copies, moves, etc.
> with the svn tools. You can't just have changes made else appear back
> in the working copy and expect them to be recognized.
Nope. We don't work on the live server. Any editing is done on the
local WC using svn add/copy/move etc. Therefore synching is a
oneway thing, from the WC => live/web server. I am not talking
about the SVN repository server which is a machine on the local
LAN.
> > but the moment you do it from a
> > different one you either have to upload *everything* (even if it is
> > 100MB) or first to equalize timestamps with those on the server to
> > get to the files different (or sync the dozen of files in as many
> > folders one by one and pray you did not miss one in the half an hour
> > that manual work takes). Maybe there are tools that handle this
> > just the other way round, but WS-FTP did not nor does
> > TotalCommander. Probably one could automize this with a post-
> > commit hook and rsync, but this also takes a lot of control out of
> > it.
>
> So do an 'rsync -av' yourself.
At first sight rsync sounds fascinating, I have had a look at it in the
past and now again, but it is not for us. We don't have Linux here
(and will not for the forseeable future, migration cost is too high,
have checked this only last year for just Word97->OpenOffice) and
running an OS layer (cygwin) on top of Windows is not very
intrigueing either. Besides until recently rsync was not provided by
the webhoster, and I think it is still not available on all of the
domains here. And the way Total Commander handles FTP synch
leaves a lot of control for me, more with far less trouble than the
commandline tool could (I use the commandline for many things).
It's a difference similar between committing with SVN commandline
and TortoiseSVN where file selection/adding, commit message, diff
are all in one dialog. The Windows rsync GUIs I just checked
OTOH seem to be either remote backup systems just happening to
use rsync protocol a/o requiring to be run on both systems as
client/server - again out of my control for the webserver side.
[text is snipped]
> >> What about comparing timestamps and filesize - if either differ
> >> then the file must be a candidate yeah? I know this is still a
> >> long way from perfect, but it would cover most of the cases that
> >> people are concerned about.
> >
> > Would not catch the corrected typo issue
>
> What's a corrected typo issue?
corrupted topy assue ==> corrected typo issue would not change
the filesize.
JH
---------------------------------------
Freedom quote:
... the question becomes,
are you going to have everyone play by the same rules,
or are you going to try to rectify the shortcomings,
errors and failures of the entire cosmos?
Because those things are wholly incompatible.
If you're going to have people play by the same rules,
that can be enforced with a minimum amount
of interference with people's freedom.
But if you're going to try to make the entire cosmos right and just,
somebody has got to have an awful lot of power
to impose what they think is right on an awful lot of other people.
What we've seen, particularly in the 20th century,
is that putting that much power in anyone's hands is enormously dangerous.
-- Thomas Sowell, 1999
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Les Mikesell <le...@gmail.com>.
Jan Hendrik wrote:
> FTP synching e.g. Not everyone has his own live server standing
> in the own basement and not every hoster provides SVN to make
> the live copy just another WC. As long as _always_ done from the
> same WC there's no issue,
Of course there are issues. You have to do adds, copies, moves, etc.
with the svn tools. You can't just have changes made else appear back
in the working copy and expect them to be recognized.
> but the moment you do it from a
> different one you either have to upload *everything* (even if it is
> 100MB) or first to equalize timestamps with those on the server to
> get to the files different (or sync the dozen of files in as many
> folders one by one and pray you did not miss one in the half an
> hour that manual work takes). Maybe there are tools that handle
> this just the other way round, but WS-FTP did not nor does
> TotalCommander. Probably one could automize this with a post-
> commit hook and rsync, but this also takes a lot of control out of it.
So do an 'rsync -av' yourself.
> Anyway, I am not going to make this a big issue, guess most
> folks here are programmers with little to no interest in websites and
> therefore having not much use for FTP sync and its pecularities
> either.
If you can rsync, why would you even think about ftp? Even if you
started with copies from a different place, rsync can romp through and
adjust any differences, including the timestamps.
>> What about comparing timestamps and filesize - if either differ then the
>> file must be a candidate yeah? I know this is still a long way from
>> perfect, but it would cover most of the cases that people are concerned
>> about.
>
> Would not catch the corrected typo issue
What's a corrected typo issue?
--
Les Mikesell
lesmikesell@gmail.com
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Jan Hendrik <ja...@myrealbox.com>.
Concerning Re: Poor performance in windows. Sw
Dmitri Colebatch wrote on 15 Feb 2007, 8:56, at least in part:
> On 2/14/07, Jing Xue <jx...@digizenstudio.com> wrote:
> >
> > I for one wouldn't care to debate on whether not updating the
> > timestamp is a user error, but I would argue strongly against paying some
> > significant performance penalty just to accommodate it - there may be
> > legitimate reasons to do that, but I can't think of any _good_ one.
FTP synching e.g. Not everyone has his own live server standing
in the own basement and not every hoster provides SVN to make
the live copy just another WC. As long as _always_ done from the
same WC there's no issue, but the moment you do it from a
different one you either have to upload *everything* (even if it is
100MB) or first to equalize timestamps with those on the server to
get to the files different (or sync the dozen of files in as many
folders one by one and pray you did not miss one in the half an
hour that manual work takes). Maybe there are tools that handle
this just the other way round, but WS-FTP did not nor does
TotalCommander. Probably one could automize this with a post-
commit hook and rsync, but this also takes a lot of control out of it.
Anyway, I am not going to make this a big issue, guess most
folks here are programmers with little to no interest in websites and
therefore having not much use for FTP sync and its pecularities
either.
> What about comparing timestamps and filesize - if either differ then the
> file must be a candidate yeah? I know this is still a long way from
> perfect, but it would cover most of the cases that people are concerned
> about.
Would not catch the corrected typo issue though it might serve for
general purpose without much (if any) of a performance penalty
while significantly enhancing SVN's guessing about a file being
modified or not. Anyway, if something like the proposed is done at
at all I think a switch option that takes *all* guessing out of it would
be by far easier - and serving the purpose much better.
JH
---------------------------------------
Freedom quote:
The only justifiable purpose of political institutions
is to assure the unhindered development of the individual.
-- Albert Einstein
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Jan Hendrik <ja...@myrealbox.com>.
Concerning Re: Poor performance in windows. Sw
Les Mikesell wrote on 15 Feb 2007, 20:59, at least in part:
> Jan Hendrik wrote:
>
> > The point, however, is that *all* FTP synching has to done from the
> > *same* WC - all the other WCs have timestamps different from those
> > on the webserver (checkout/commit time).
>
> I'd be even more extreme about this and use that WC only for staging.
> That is, never edit or commit from there. Instead, edit/commit from
> other working copies then update this WC, do any final testing you
> might do, and push to production. That not only avoids timestamp
> issues, but it eliminates the possibility of putting something in
> production that you had changed locally and not committed - which can
> lead to not being able to back out the next change, should the need
> arise.
Yep, that would be kind of optimum. Incidentally some days ago I
had some thoughts in this direction, too: a WC exclusively
updated and rsynced with webserver through post-commit hook,
but there are issues with this as on this LAN no machine is
guaranteed to be on at any time, not even the one running the SVN
repository, move/rename in SVN could lead to non-working states
on webserver because taking two commits, files generated on the
server only might get deleted, generally too little control (not being
aware then of the cygwin layer needed for rsync I am quite
contempt about though I will do some further reading on it again if
time allows).
Generally the way we do it currently comes as close as possible.
We had a Linkstation as NAS fileserver, but updating a WC on it
was arduous as the SVN client not runs on the Linkstation itself
naturally. With 95% of stuff being content editing/adding with
code&link checks already done before commit using a regular WC
for webserver is not too dangerous. Basically timestamp issues
arise if things get into a hurry, a machine becomes unavailable, etc.
JH
---------------------------------------
Freedom quote:
In the end commerce and trade are the sources of your wealth
and these sources are constituted so
that who scoops from them cannot enrich himself
without making many other rich as well.
-- Montesquieu
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Les Mikesell <le...@gmail.com>.
Jan Hendrik wrote:
>> That shouldn't be too much of a problem, since you should be
>> committing to SVN before uploading to the FTP anyway.
>
> "Should" is a very good word in this instance - usually such things
> happen if there is a hurry. <G> Just think about this: you edit
> some content and suddenly notice the price lacks a zero at the
> end. Your editing isn't ready for commit, but this price must be
> corrected immediately on the live server, even if this results in a
> half-edited page for a couple of hours.
>
If reliability and having everything version-controlled matters to you,
make it so the only way into production is through the repository. Then
make it so it isn't a lot harder. If you don't have a real QA/testing
team and procedure it can be as simple as a batch file that commits from
the WC where you make changes, then changes directory to another WC used
for staging on the same machine, does an update and transfers those
files to production. This makes sure that you pick up any work done and
committed elsewhere and that if any change causes a problem you can
update the staging directory to a previously known-working revision to
fix things quickly. You might or might not want to bother copying to
tags to track the versions that have been pushed to production.
--
Les Mikesell
lesmikesell@gmail.com
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
RE: Poor performance in windows. Switching back to CVS
Posted by Jan Hendrik <ja...@myrealbox.com>.
Concerning RE: Poor performance in windows. Sw
Gavin Lambert wrote on 16 Feb 2007, 13:45, at least in part:
> Quoth Jan Hendrik <ma...@myrealbox.com>:
> > The point, however, is that *all* FTP synching has to done from the
> > *same* WC - all the other WCs have timestamps different from those
> > on the webserver (checkout/commit time).
> >
> > So if for any reason an FTP synch is to be done from another WC
> > local timestamps first have to be adjusted to remote timestamps or
> > FTP synch would consider *all* files as modified. Afterwards FTP
> > will still indicate files that are different (more exactly: that
> > have a different filesize), but SVN will not commit any file anymore
> > that was not committed before FTP synch, no matter if the filesize
> > had changed or not (the typo thing where filesize does not change
> > either is a special case).
>
> That shouldn't be too much of a problem, since you should be
> committing to SVN before uploading to the FTP anyway.
"Should" is a very good word in this instance - usually such things
happen if there is a hurry. <G> Just think about this: you edit
some content and suddenly notice the price lacks a zero at the
end. Your editing isn't ready for commit, but this price must be
corrected immediately on the live server, even if this results in a
half-edited page for a couple of hours.
> If you want to persist in this behaviour, then I'd recommend the
> following strategy instead:
Thanks for the recipe, Gavin. I shall pin it to each WC.
[...]
> (Besides, using timestamps as your only means of detecting
> modifications is fragile, which is why SVN doesn't do it -- what
> happens if one computer has the wrong date set?)
Oops, hasn't the wheel come full circle here somehow with SVN
firsthand relying on timestamps? Sure, chances are certainly
small that a wrong date would lead to a modified stamp equal to
the pre-modified one, but chances are. One does not even need to
take a file to another machine, just play with Windows clock
properties a bit (without any saving!) and then abort - part of the
things are taken over just the same.
JH
---------------------------------------
Freedom quote:
The object and practice of liberty lies
in the limitation of governmental power.
-- Gen. Douglas MacArthur
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Les Mikesell <le...@gmail.com>.
Jan Hendrik wrote:
> The point, however, is that *all* FTP synching has to done from the
> *same* WC - all the other WCs have timestamps different from
> those on the webserver (checkout/commit time).
I'd be even more extreme about this and use that WC only for staging.
That is, never edit or commit from there. Instead, edit/commit from
other working copies then update this WC, do any final testing you might
do, and push to production. That not only avoids timestamp issues, but
it eliminates the possibility of putting something in production that
you had changed locally and not committed - which can lead to not being
able to back out the next change, should the need arise.
--
Les Mikesell
lesmikesell@gmail.com
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Jan Hendrik <ja...@myrealbox.com>.
Concerning Re: Poor performance in windows. Sw
Jeff Smith wrote on 16 Feb 2007, 7:47, at least in part:
> I suppose Jan means they all FTP-sync to one group server which has
> the svn client... Howabout rather than having only one "group WC" on
> that server, everyone have a seperate WC there so each syncs only with
> their own. Maybe not since one WC has a huge number of files.
>
> Well, to me, this keeps comming back to two rules for using
> subversion, which should never be impossible to meet:
>
> 1. Each developer must have their own svn WC (subversion working
> copy). I still saw no reason why you cannot each have, on your own PC,
> an svn client which can be secured just as much as any other form of
> file transfer.
They have, each on their own PC, one of the PCs also acting as
SVN server for the repository.
It's the production webserver that is completely out of reach for
SVN and that we have to get stuff synched onto via FTP.
> 2. Each developer should only check out the part of the project they
> are working on. At least I have never seen a project where each member
> of the team must constantly work on 10,000 files at once. They have
> always been broken up into categories (by subdir) so that one week I
> might be working only on the "drivers/wheel" component.
We are not doing development, but web content (catalog of stock
items). That's kind of different workflow. And we do this just for
own needs. Although it is broken up into categories 2-3 levels
deep they all are more or less loosely bound to each other.
Usually there is nothing I would work on for a week or even a day
either, and usually there happens to be a cross-link necessary
here and there. We would spend our days checking out & in and
making tons of notes for links to be set after the next checkout of
this or that category. :)
I'll see if rsync will bring us forth anyhow.
JH
---------------------------------------
Freedom quote:
American history has proven when people live in freedom
the vast majority of sovereign individuals believe in,
and practice, the golden rule.
-- Rick Gaber
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Jeff Smith <js...@robotronics.com>.
On Thursday 15 February 2007 20:31, Les Mikesell wrote:
> Jan Hendrik wrote:
> > The point, however, is that *all* FTP synching has to done from
> > the *same* WC - all the other WCs have timestamps different from
> > those on the webserver (checkout/commit time).
>
> I'd be even more extreme about this and use that WC only for
> staging. That is, never edit or commit from there. Instead,
> edit/commit from other working copies then update this WC, do any
> final testing you might do, and push to production. That not only
> avoids timestamp issues, but it eliminates the possibility of
> putting something in production that you had changed locally and
> not committed - which can lead to not being able to back out the
> next change, should the need arise.
I suppose Jan means they all FTP-sync to one group server which has
the svn client... Howabout rather than having only one "group WC" on
that server, everyone have a seperate WC there so each syncs only
with their own. Maybe not since one WC has a huge number of files.
Well, to me, this keeps comming back to two rules for using
subversion, which should never be impossible to meet:
1. Each developer must have their own svn WC (subversion working
copy). I still saw no reason why you cannot each have, on your own
PC, an svn client which can be secured just as much as any other form
of file transfer.
2. Each developer should only check out the part of the project they
are working on. At least I have never seen a project where each
member of the team must constantly work on 10,000 files at once. They
have always been broken up into categories (by subdir) so that one
week I might be working only on the "drivers/wheel" component.
Planning the structure like this is crucial, esp. if we know we will
have a huge number of files to manage.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
RE: Poor performance in windows. Switching back to CVS
Posted by Gavin Lambert <ga...@compacsort.com>.
Quoth Jan Hendrik <ma...@myrealbox.com>:
> The point, however, is that *all* FTP synching has to done from the
> *same* WC - all the other WCs have timestamps different from
> those on the webserver (checkout/commit time).
>
> So if for any reason an FTP synch is to be done from another WC
> local timestamps first have to be adjusted to remote timestamps or
> FTP synch would consider *all* files as modified. Afterwards FTP
> will still indicate files that are different (more exactly:
> that have a
> different filesize), but SVN will not commit any file anymore that
> was not committed before FTP synch, no matter if the filesize had
> changed or not (the typo thing where filesize does not change
> either is a special case).
That shouldn't be too much of a problem, since you should be committing
to SVN before uploading to the FTP anyway.
If you want to persist in this behaviour, then I'd recommend the
following strategy instead:
1. When FTP Sync is used to upload the files to the FTP server, note
down the revision number of the working copy at the time.
2. When you want to update the files on the server, and for whatever
reason you must use a different working copy, then do this:
2a. Make sure you've committed all changes that you've made
2b. Update your working copy to the *same* revision as was uploaded to
the FTP server
2c. Use your FTP Sync program to set all local timestamps equal to the
remote ones
2d. Update the working copy to the revision you want to upload (probably
HEAD)
2e. Use your FTP Sync program to do the actual upload
The update in #2d might take a little while, since it'll see the wrong
timestamps and have to do a content scan, but at the end of it you
should get a working copy where the timestamps differ only for modified
files. At least as long as the update doesn't reset the timestamps on
unmodified files, but I don't think it does.
But I basically agree with Jeff. Trying to use two different version
controlling methods on the same directory would get fairly painful.
(Besides, using timestamps as your only means of detecting modifications
is fragile, which is why SVN doesn't do it -- what happens if one
computer has the wrong date set?)
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Jan Hendrik <ja...@myrealbox.com>.
Concerning Re: Poor performance in windows. Sw
Jeff Smith wrote on 15 Feb 2007, 10:01, at least in part:
> On Thursday 15 February 2007 04:45, Jan Hendrik wrote:
> > FTP synching e.g. Not everyone has his own live server standing in
> > the own basement and not every hoster provides SVN to make the live
> > copy just another WC. As long as _always_ done from the same WC
> > there's no issue, but the moment you do it from a
> [...]
>
> My first thought was, what does FTP synching have to do with
> Subversion? Do you mean that this is where you are getting your
> timestamps messed up, and that's why you need a "--ignore-timestamp"
> option? In that case I agree, the option is needed for quick
> workarounds. At the same time, it would not be a solution to the
> problem. I'd suggest fixing your messed-up FTP sync.
Think I mentioned in a previous posting how I happened to get into
the timestamp trap, but here step by step (please accept my
appologies for this becoming lengthier):
1) Web site, approx. 10K files content (HTML & JPG; semi-static,
actually PHP, headers, footers, etc. included. Running this out of
a database is something I would rather not for a couple of reasons,
this as a sidenote)
2) The files are worked on on a small LAN, three machines (W2K),
four WCs.
3) The repository is on one of the machines on the LAN, no access
from outside (as much as I hope! <GG>)
4) The webspace is hosted outside. Access: telnet, SSH, FTP
5) Files are uploaded to the webserver through FTP using one of
the WCs as local source. Long ago we used WS-FTP, then
changed to Total Commander. AFAICT they both work the same:
compare local & remote folder trees by checking timestamps &
filesize, indicating files that are different in either one or both. Now
in synch mode local files are uploaded, then *local* timestamps
are set to *remote* timestamps so the next run these files would
not be listed for upload again. AFAIK this is the way FTP works -
the files are not copied, but their content written into a (new) file on
the target.
So far FTP synching has nothing to do with and does not interfere
with version control. That synching changes the local timestamps
results in some performance penalty with SVN for the respective
WC, but running TSVN cleanup now and then takes care of this.
Though it involves some manual activity the whole process leaves
the control to me - files can be excluded at will, files generated on
the web server can be taken care of, and I can see immediately if
something has not worked correctly.
The point, however, is that *all* FTP synching has to done from the
*same* WC - all the other WCs have timestamps different from
those on the webserver (checkout/commit time).
So if for any reason an FTP synch is to be done from another WC
local timestamps first have to be adjusted to remote timestamps or
FTP synch would consider *all* files as modified. Afterwards FTP
will still indicate files that are different (more exactly: that have a
different filesize), but SVN will not commit any file anymore that
was not committed before FTP synch, no matter if the filesize had
changed or not (the typo thing where filesize does not change
either is a special case).
So not FTP synch is messed up, it's that SVN and FTP synch
don't go together too well. We could leave it at that if we had not
found several other cases timestamps can get mixed up to a point
SVN ignores modified files (and thus risks losing data). Neither
would it be a good argument to say Then use something else than
FTP. No application should limit one's choice for the rest of
programs in the whole workflow (remember the Trash your editor
argument in the mixed-EOL thread?) beyond cases of severe
inevitable interference.
> It might help to know how to use Subversion appropriately to sync your
> personal repository with another. Speaking from experience, you don't
> need a separate server in your basement, honest. Any PC these days is
> a PS (Personal Server), even the same one you do work on.
In the case here it's not about synching two repositories. It's just
about keeping a remote webserver in synch with the local copy that
is an SVN WC while the remote one is not and cannot be (neither
is SVN setup on the webserver nor do I have the option to set it up
there nor would I want to do so just for the hassle of opening a LAN
to remote access not needed for anything else).
> > > What about comparing timestamps and filesize - if either differ
> > > then the file must be a candidate yeah? I know this is still a
> > > long way from perfect, but it would cover most of the cases that
> > > people are concerned about.
> >
> > Would not catch the corrected typo issue though it might serve for
> > general purpose without much (if any) of a performance penalty while
> > significantly enhancing SVN's guessing about a file being modified
> > or not. Anyway, if something like the proposed is done at at all I
> > think a switch option that takes *all* guessing out of it would be
> > by far easier - and serving the purpose much better.
> >
> > JH
>
> "corrected typo"... CORRECT TYPO?? So you're the culprit! Correcting
> a typo is definately changing content. Honestly, how can you claim
> that ANY type of search&replace operations could justify preserving
> the mod timestamp!?
Alright. How I happened to come about this I have tried to explain
above. We had some other considerations about how this situation
can happen (taking a file - not the WC - to another machine with a
clock off synch and copying it back into one's WC the other day
e.g.) and I also have seen and probably used in the past tools for
S&R with the explicit option to preserve timestamps, whatever the
use or idea of this may be. Certainly one can argue that taking a
file out of a WC for homework on a machine with a clock out of
synch and putting the file back into the WC on Monday is a user
error - the user should have checked the clock, the timestamps,
downloaded 100MB WC to his home machine, should not use a
legitimate *nix command like touch, etc. etc., but finally computers
and applications shall work without the user having to go to the
moon, shouldn't they?
Sorry that this has grown that much out of what was just a side-
note, didn't intend to spend so much time on it either. For me it is
an annoyance I usually can work around by strictly using one and
only one specific WC for webserver synching. Nevertheless it
reveals kind of weakness or flaw in SVN, a hole where data loss
can happen, and it would be nice if this could be closed by the
suggested switch. Personally much more annoying that I had to
take Python code out of versioning because of SVN not merging it,
but creating conflicts where no conflicts were. That is a different
issue, however.
JH
---------------------------------------
Freedom quote:
Intellect annuls Fate. So far as a man thinks, he is free ...
The revelation of Thought takes man out of servitude into freedom.
-- Ralph Waldo Emerson, Fate
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Jeff Smith <js...@robotronics.com>.
On Thursday 15 February 2007 04:45, Jan Hendrik wrote:
> FTP synching e.g. Not everyone has his own live server standing
> in the own basement and not every hoster provides SVN to make
> the live copy just another WC. As long as _always_ done from the
> same WC there's no issue, but the moment you do it from a
[...]
My first thought was, what does FTP synching have to do with
Subversion? Do you mean that this is where you are getting your
timestamps messed up, and that's why you need a "--ignore-timestamp"
option? In that case I agree, the option is needed for quick
workarounds. At the same time, it would not be a solution to the
problem. I'd suggest fixing your messed-up FTP sync.
It might help to know how to use Subversion appropriately to sync your
personal repository with another. Speaking from experience, you don't
need a separate server in your basement, honest. Any PC these days is
a PS (Personal Server), even the same one you do work on.
> > What about comparing timestamps and filesize - if either differ
> > then the file must be a candidate yeah? I know this is still a
> > long way from perfect, but it would cover most of the cases that
> > people are concerned about.
>
> Would not catch the corrected typo issue though it might serve for
> general purpose without much (if any) of a performance penalty
> while significantly enhancing SVN's guessing about a file being
> modified or not. Anyway, if something like the proposed is done at
> at all I think a switch option that takes *all* guessing out of it
> would be by far easier - and serving the purpose much better.
>
> JH
"corrected typo"... CORRECT TYPO?? So you're the culprit! Correcting
a typo is definately changing content. Honestly, how can you claim
that ANY type of search&replace operations could justify preserving
the mod timestamp!?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
RE: Poor performance in windows. Switching back to CVS
Posted by Gavin Lambert <ga...@compacsort.com>.
Quoth Erik Huelsmann <ma...@gmail.com>:
> Well, not entirely: there are still many cases which wouldn't
> be covered by your criteria: moving code around for example doesn't
> cause a size change.
Yeah, but if people are using broken editing tools then the correct
response is to use different tools.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Erik Huelsmann <eh...@gmail.com>.
On 2/14/07, Dmitri Colebatch <di...@colebatch.com> wrote:
> On 2/14/07, Jing Xue <jx...@digizenstudio.com> wrote:
> > I for one wouldn't care to debate on whether not updating the
> > timestamp is a user error, but I would argue strongly against paying some
> > significant performance penalty just to accommodate it - there may be
> > legitimate reasons to do that, but I can't think of any _good_ one.
>
>
> What about comparing timestamps and filesize - if either differ then the
> file must be a candidate yeah?
Sure.
> I know this is still a long way from
> perfect, but it would cover most of the cases that people are concerned
> about.
Well, not entirely: there are still many cases which wouldn't be
covered by your criteria: moving code around for example doesn't cause
a size change.
bye,
Erik.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Dmitri Colebatch <di...@colebatch.com>.
On 2/14/07, Jing Xue <jx...@digizenstudio.com> wrote:
>
> I for one wouldn't care to debate on whether not updating the
> timestamp is a user error, but I would argue strongly against paying some
> significant performance penalty just to accommodate it - there may be
> legitimate reasons to do that, but I can't think of any _good_ one.
What about comparing timestamps and filesize - if either differ then the
file must be a candidate yeah? I know this is still a long way from
perfect, but it would cover most of the cases that people are concerned
about.
cheers
dim
Re: Poor performance in windows. Switching back to CVS
Posted by Jing Xue <jx...@digizenstudio.com>.
On Tue, Feb 13, 2007 at 01:53:10PM -0700, Jeff Smith wrote:
>
> Shoot... are we trying to make svn perform faster, or grind the
> process to a halt??
>
> Can you imagine the increased _drag___ if svn were supposed to assume
> that every timestamp were meaningless, and have to compare entire
> contents instead? That would be horrendous! I say if you haven't got
> valid timestamps (especially for large number of files), you haven't
> got version control.
+1.
I for one wouldn't care to debate on whether not updating the
timestamp is a user error, but I would argue strongly against paying some
significant performance penalty just to accommodate it - there may be
legitimate reasons to do that, but I can't think of any _good_ one.
--
Jing Xue
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Jan Hendrik <ja...@myrealbox.com>.
Concerning Re: Poor performance in windows. Sw
Jeff Smith wrote on 13 Feb 2007, 13:53, at least in part:
> On Tuesday 13 February 2007 02:31, Jan Hendrik wrote:
> > Well, is it really to be considered a user error? There are more
> > than one apps out there that deliberately or by user's choice do not
> > touch the timestamp on search&replace operations for one or other
> > reason.
>
> Shoot... are we trying to make svn perform faster, or grind the
> process to a halt??
>
> Can you imagine the increased _drag___ if svn were supposed to assume
> that every timestamp were meaningless, and have to compare entire
> contents instead? That would be horrendous!
Well, make it a case of --force switch as I said downwards of that
or probably a second posting!
> I say if you haven't got
> valid timestamps (especially for large number of files), you haven't
> got version control.
Dunno what timestamps (any unchanged timestamp still is a valid
timestamp if the filesystem is not broken) have to do with version
control. I would rather having SVN optionally honoring last modified
as timestamp for checkouts/updates instead of
commit/checkout/update time anyway. Timestamps may not
matter in C programming, with FTP synching they matter.
JH
---------------------------------------
Freedom quote:
Freedom is the power to live as you will.
Who then lives as he wills?
-- Marcus Tullius Cicero
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Jeff Smith <js...@robotronics.com>.
On Tuesday 13 February 2007 02:31, Jan Hendrik wrote:
> Well, is it really to be considered a user error? There are more
> than one apps out there that deliberately or by user's choice do
> not touch the timestamp on search&replace operations for one or
> other reason.
Shoot... are we trying to make svn perform faster, or grind the
process to a halt??
Can you imagine the increased _drag___ if svn were supposed to assume
that every timestamp were meaningless, and have to compare entire
contents instead? That would be horrendous! I say if you haven't got
valid timestamps (especially for large number of files), you haven't
got version control.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Ryan Schmidt <su...@ryandesign.com>.
On Feb 12, 2007, at 15:31, Jan Hendrik wrote:
> OH, BTW there is a bug in SVN: if a file is modified without
> changing timestamp and/or filesize, SVN will not commit the file,
> and I think if the file is updated local changes are lost.
This behavior does exist in Subversion, but I do not believe it is
considered a bug. Rather, I believe it is considered a user error to
modify a file but not update its timestamp.
--
To reply to the mailing list, please use your mailer's Reply To All
function
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Jan Hendrik <ja...@myrealbox.com>.
Concerning Re: Poor performance in windows. Sw
John Allen wrote on 12 Feb 2007, 21:00, at least in part:
> B. Smith-Mannschott wrote:
[text is snipped]
> > the OP's issue isn't performance on the server side. The issue is
> > client side (working copy) performance. Particularly for operations
> > that have to thumb through a *lot* of files, like update.
> >
> Update should not be "thumbing" through a lot of files, the svn server
> should supply the diffs between
> your working copy revision, and the current revision on the server.
>
> Now commits are a different story, the client has to scan the working
> copy to find the changes you
> have made, and this will be slow on Windows if you have 1000's files in
> your working copy.
Huh? A web project with roughly 11,000 files in about 300 folders
(33,000 files in 1300 folders including all .svn meta-overhead)
updates & commits quite fast on a 1.6Centrino machine, either
W2K SP4 or XP SP2, NTFS, SVN 1.4.2, using TSVN 1.4.1, and
still reasonably on a P200 machine (TSVN commit dialog still
within 30 secs). Repository (BDB) is on a 1.5 P4, also W2K SP4,
NTFS, SVN 1.4.2, access through Apache 2.0.59. Getting the
logs takes a bit longer, also on the Centrino box. And the actual
update/commit time of course depends on the number of files
involved.
Maybe the OP should turn off Windows indexing (I think he already
did), defragment his HDD, clean up the WC (probably using TSVN
as this reportedly resets the timestamps in working base in case
they have changed in WC without the files having changed; dunno if
SVN commandline does this, too.
OH, BTW there is a bug in SVN: if a file is modified without
changing timestamp and/or filesize, SVN will not commit the file,
and I think if the file is updated local changes are lost.
JH
---------------------------------------
Freedom quote:
Intellect annuls Fate. So far as a man thinks, he is free ...
The revelation of Thought takes man out of servitude into freedom.
-- Ralph Waldo Emerson, Fate
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Duncan Murdoch <mu...@stats.uwo.ca>.
On 2/14/2007 3:20 AM, L. Wayne Johnson wrote:
>
>>> -----Original Message-----
>>> From: Gavin Lambert [mailto:gavinl@compacsort.com]
>>> Sent: Tuesday, February 13, 2007 3:26 PM
>>> To: 'Phillip Susi'; 'Ryan Schmidt'
>>> Cc: users@subversion.tigris.org
>>> Subject: RE: Poor performance in windows. Switching back to CVS
>>>
>>> Quoth Phillip Susi <ma...@cfl.rr.com>:
>>> > If the timestamps get messed up ( and I used to see this
>>> > frequently on windows ) then all the files must be read in
>>> > full, causing the process to slow to a crawl. An svn cleanup
>>> > should fix that.
>>>
>>> Actually, on that note: if the OP has their working copy on a FAT/FAT32
>>> drive then this could cause problems. FAT only stores filestamps with a
>>> 2-second resolution, so unless software that compares timestamps knows
>>> that and specifically ignores one-second differences in the timestamp
>>> then they'll think the files have changed even when they haven't, which
>>> in SVN's case would lead to performance loss as it needs to rescan the
>>> contents of each affected file.
>>>
>>> I don't know if SVN ignores one-second differences in timestamps or not,
>>> so I don't know if it's vulnerable to this.
>
> If they are using FAT and long file names the problem is significantly worse
> than the time stamp. Long file names use multiple directory entries per
> name, the number depends on the length of the name. Not only can you get
> file fragmentation but you can get file name fragmentation as well. As files
> with different lengths names are deleted and created the names become
> interleaved and the pieces of the name can get rather far apart possible in
> a different block...
>
> Theoretically you can end up traversing the entire directory listing just to
> get a single file name.
That's not completely correct. The directory entries for one file are
always contiguous once you've put all the directory clusters together
into one big file. However, it is possible to start in one directory
cluster and continue at the start of the next one, and the free space
can become fragmented, so directories tend to grow over time: if a
filename needs 5 entries to hold it, no combination of smaller pieces
will do.
Duncan Murdoch
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
RE: Poor performance in windows. Switching back to CVS
Posted by "L. Wayne Johnson" <wa...@zk.com>.
>> -----Original Message-----
>> From: Gavin Lambert [mailto:gavinl@compacsort.com]
>> Sent: Tuesday, February 13, 2007 3:26 PM
>> To: 'Phillip Susi'; 'Ryan Schmidt'
>> Cc: users@subversion.tigris.org
>> Subject: RE: Poor performance in windows. Switching back to CVS
>>
>> Quoth Phillip Susi <ma...@cfl.rr.com>:
>> > If the timestamps get messed up ( and I used to see this
>> > frequently on windows ) then all the files must be read in
>> > full, causing the process to slow to a crawl. An svn cleanup
>> > should fix that.
>>
>> Actually, on that note: if the OP has their working copy on a FAT/FAT32
>> drive then this could cause problems. FAT only stores filestamps with a
>> 2-second resolution, so unless software that compares timestamps knows
>> that and specifically ignores one-second differences in the timestamp
>> then they'll think the files have changed even when they haven't, which
>> in SVN's case would lead to performance loss as it needs to rescan the
>> contents of each affected file.
>>
>> I don't know if SVN ignores one-second differences in timestamps or not,
>> so I don't know if it's vulnerable to this.
If they are using FAT and long file names the problem is significantly worse
than the time stamp. Long file names use multiple directory entries per
name, the number depends on the length of the name. Not only can you get
file fragmentation but you can get file name fragmentation as well. As files
with different lengths names are deleted and created the names become
interleaved and the pieces of the name can get rather far apart possible in
a different block...
Theoretically you can end up traversing the entire directory listing just to
get a single file name.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
RE: Poor performance in windows. Switching back to CVS
Posted by Gavin Lambert <ga...@compacsort.com>.
Quoth Phillip Susi <ma...@cfl.rr.com>:
> If the timestamps get messed up ( and I used to see this
> frequently on windows ) then all the files must be read in
> full, causing the process to slow to a crawl. An svn cleanup
> should fix that.
Actually, on that note: if the OP has their working copy on a FAT/FAT32
drive then this could cause problems. FAT only stores filestamps with a
2-second resolution, so unless software that compares timestamps knows
that and specifically ignores one-second differences in the timestamp
then they'll think the files have changed even when they haven't, which
in SVN's case would lead to performance loss as it needs to rescan the
contents of each affected file.
I don't know if SVN ignores one-second differences in timestamps or not,
so I don't know if it's vulnerable to this.
NTFS uses a much higher resolution for timestamps, so would not be
affected by this problem.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Phillip Susi <ps...@cfl.rr.com>.
Ryan Schmidt wrote:
> But both the update and the commit *do* need to thumb through the entire
> working copy to read through all the .svn directories to find out what
> the current revision of all the files and directories is.
They only need to read the control files and check the timestamps for
most of the files. This is no different than cvs, so it does not
account for a slowdown with svn.
If the timestamps get messed up ( and I used to see this frequently on
windows ) then all the files must be read in full, causing the process
to slow to a crawl. An svn cleanup should fix that.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Ryan Schmidt <su...@ryandesign.com>.
On Feb 12, 2007, at 15:00, John Allen wrote:
>> the OP's issue isn't performance on the server side. The issue is
>> client side (working copy) performance. Particularly for
>> operations that have to thumb through a *lot* of files, like update.
>
> Update should not be "thumbing" through a lot of files, the svn
> server should supply the diffs between
> your working copy revision, and the current revision on the server.
>
> Now commits are a different story, the client has to scan the
> working copy to find the changes you
> have made, and this will be slow on Windows if you have 1000's
> files in your working copy.
But both the update and the commit *do* need to thumb through the
entire working copy to read through all the .svn directories to find
out what the current revision of all the files and directories is.
--
To reply to the mailing list, please use your mailer's Reply To All
function
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by John Allen <jo...@dublinux.net>.
B. Smith-Mannschott wrote:
>
> On Feb 12, 2007, at 16:44, Phillip Susi wrote:
>
>> Julian Hsiao wrote:
>>> In article <45...@atlantico.com.br>,
>>> Joaquim Oliveira <jo...@atlantico.com.br> wrote:
>>>> I searched the mail list archives, but couldn't find a solution for
>>>> this. I found something about "the NTFS file system does not
>>>> perform well when you have a large number of small files", but we
>>>> need to develop in Windows, so adopting Linux/Ext3 is not an
>>>> option. I've already seen these messages:
>
> note: *develop in Windows*
>
>>> There are kernel drivers for 2k/XP for ext2. If the bottleneck is
>>> indeed NTFS, perhaps you can create a ext2 partition specifically
>>> for your working copy.
>>> It's worth a shot, no?
>>
>> Or you can use a db backend, which uses far fewer files, instead of
>> the fs backend.
>>
>
> the OP's issue isn't performance on the server side. The issue is
> client side (working copy) performance. Particularly for operations
> that have to thumb through a *lot* of files, like update.
>
Update should not be "thumbing" through a lot of files, the svn server
should supply the diffs between
your working copy revision, and the current revision on the server.
Now commits are a different story, the client has to scan the working
copy to find the changes you
have made, and this will be slow on Windows if you have 1000's files in
your working copy.
>> And just because you develop on windows does not mean your server
>> must run windows. I develop an embedded system on windows because
>> that is where the vendor tools are supplied, but I migrated our
>> server to linux to host the svn repo and bug tracking database,
>> because these tools simply run better under linux and the server is
>> more reliable too.
>
> // ben
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: users-help@subversion.tigris.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by "B. Smith-Mannschott" <be...@gmail.com>.
On Feb 12, 2007, at 16:44, Phillip Susi wrote:
> Julian Hsiao wrote:
>> In article <45...@atlantico.com.br>,
>> Joaquim Oliveira <jo...@atlantico.com.br> wrote:
>>> I searched the mail list archives, but couldn't find a solution
>>> for this. I found something about "the NTFS file system does not
>>> perform well when you have a large number of small files", but we
>>> need to develop in Windows, so adopting Linux/Ext3 is not an
>>> option. I've already seen these messages:
note: *develop in Windows*
>> There are kernel drivers for 2k/XP for ext2. If the bottleneck is
>> indeed NTFS, perhaps you can create a ext2 partition specifically
>> for your working copy.
>> It's worth a shot, no?
>
> Or you can use a db backend, which uses far fewer files, instead of
> the fs backend.
>
the OP's issue isn't performance on the server side. The issue is
client side (working copy) performance. Particularly for operations
that have to thumb through a *lot* of files, like update.
> And just because you develop on windows does not mean your server
> must run windows. I develop an embedded system on windows because
> that is where the vendor tools are supplied, but I migrated our
> server to linux to host the svn repo and bug tracking database,
> because these tools simply run better under linux and the server is
> more reliable too.
// ben
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Phillip Susi <ps...@cfl.rr.com>.
Julian Hsiao wrote:
> In article <45...@atlantico.com.br>,
> Joaquim Oliveira <jo...@atlantico.com.br> wrote:
>
>> I searched the mail list archives, but couldn't find a solution for
>> this. I found something about "the NTFS file system does not perform
>> well when you have a large number of small files", but we need to
>> develop in Windows, so adopting Linux/Ext3 is not an option. I've
>> already seen these messages:
>
> There are kernel drivers for 2k/XP for ext2. If the bottleneck is indeed
> NTFS, perhaps you can create a ext2 partition specifically for your working
> copy.
>
> It's worth a shot, no?
>
Or you can use a db backend, which uses far fewer files, instead of the
fs backend.
And just because you develop on windows does not mean your server must
run windows. I develop an embedded system on windows because that is
where the vendor tools are supplied, but I migrated our server to linux
to host the svn repo and bug tracking database, because these tools
simply run better under linux and the server is more reliable too.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Julian Hsiao <ma...@nyanisore.net>.
In article <45...@atlantico.com.br>,
Joaquim Oliveira <jo...@atlantico.com.br> wrote:
> I searched the mail list archives, but couldn't find a solution for
> this. I found something about "the NTFS file system does not perform
> well when you have a large number of small files", but we need to
> develop in Windows, so adopting Linux/Ext3 is not an option. I've
> already seen these messages:
There are kernel drivers for 2k/XP for ext2. If the bottleneck is indeed
NTFS, perhaps you can create a ext2 partition specifically for your working
copy.
It's worth a shot, no?
Julian Hsiao
madoka@nyanisore.net
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Jared Hardy <ja...@gmail.com>.
On 2/6/07, Joaquim Oliveira <jo...@atlantico.com.br> wrote:
> Our team tried svn for 4 months, but we're switching back to CVS due to
> performance problems.
I feel your pain. :) Our project actually started out using WinCVS,
but then switched to Subversion. The main reason we would never switch
back is that we version large binaries (3D art and related files), and
require a lock system for preventing work loss from files that can't
be merged. We only started seeing a lot of slowdown when certain
folders started having 512 or more sub-folders only one level deep --
this seems to be an implementation limit in NTFS (and CIFS), as all
interaction with such folders slows down all versions of Windows, both
in and out of Subversion use.
We also version built files, so we were constantly annoyed with
unnecessary "unversioned file of the same name already exists" errors
on update and commit. We had to script around that.
> SVN access was made using svn:// protocol. CVS was faster in all
> situations. Most of then, it is twice faster. For example, an update in
> working copy root folder was about 20% faster in CVS.
I find this strange, because if I remember correctly, .cvs working
copy cache folders are strewn all over the working copy, just like
.svn folders. That is the biggest bottleneck with update and commit
actions for us. I find that SCM systems with central Working Copy
cache stores, including Perforce and SVK, all work *much faster* on
NTFS.
> We noticed that SVN creates more administrative files and directories
> than CVS. The checkout size is:
> - CVS: 24849 files, 4841 folders. Disk usage: 164 MB
> - SVN: 27450 files, 22319 folders (!). Disk usage: 261 MB
Is disk usage a big issue for you, or are you just noting this as
a potential bottleneck? If you have the disk space to spare, I highly
recommend SVK. With SVK 2.0.0 and a Subversion 1.4.2 FSFS repository
on each client, we are getting updates 50x to 1000x faster (depending
on number and breadth of changes), and commits 10x faster. The SVK
FSFS repository is taking about 50GB, due to a deep history of over
26K revisions, which is about 4x as much as a standard .svn WC cache
would take. The time and network bandwidth savings are much more
important to us, than ~$15 worth of extra disk space used per
workstation.
The main drawbacks to SVK are lack of a "svk lock" command, and lack
of GUI clients. We are working around both of those limits with
internal tools and scripts for now. We are also working on an "edit"
command and cache system, similar to a Perforce "checkout", to make
the commit times closer to instantaneous.
> I searched the mail list archives, but couldn't find a solution for
> this. I found something about "the NTFS file system does not perform
> well when you have a large number of small files"
The way NTFS stores meta-data, like timestamps and file size, is
horribly inefficient for large folder structures, and these meta-data
statistics are all analyzed repeatedly during update and commit
cycles. The way .svn and .cvs WC cache files are strewn throughout the
folders doesn't help matters. Matching something like SVK, with a
central WC cache, and a file system with efficient meta-data storage,
like ReiserFS, would be the best of both worlds. With NTFS, your best
hope is a central WC cache -- at least it wont make the recursive
folder-tree walks any slower.
> Is there any way to improve SVN performance? What are the most common
> bottlenecks?
Like I said, keep 1-level deep folder counts to below 512 on NTFS. IFS
might have the same implementation issue, so it's not clear if using a
different file system would help on Win32. Defrag often. Keeping the
server and all clients to SVN 1.4.2+ seems to help a little, in terms
of delta transfer time, and bandwidth used. The SVN 1.4 WC cache is
slightly faster, but it still has the problem of .svn folder
pollution.
If you can, use SVK. If Subversion has the best available
repository model, then SVK has the best available Working Copy cache
model. Using SVK with a CVS repository is a lot more difficult under
Win32 right now, but possible. If you use any central WC cache system,
like SVK, keeping the WC cache/repository on one disk (slower but
bigger), and your work area on another disk (faster writes, like a 10K
rpm disk), improves speed. Disk to disk tranfers are almost always
faster than disk to self copies.
:) Jared
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Talden <ta...@gmail.com>.
For the developers, ensure that their virus scanner doesn't
excessively scan the working copy (at the least ignoring .svn content,
at best ignore the whole working copy) and as already noted don't
allow windows to index these files.
For CVS and Subversion I'd expect both of these changes to pay
dividends in Windows file-system performance (well as much as the
windows file-system has to offer anyway).
--
Talden
On 2/8/07, Joaquim Oliveira <jo...@atlantico.com.br> wrote:
> Dmitri Colebatch escreveu:
> > Do you guys tag at all? We've recently migrated a 16000 odd file
> > repository and have certainly found that some operations are slower,
> > but the tagging speed of svn compared to cvs is something that would
> > definately stop me from switching back (o:
> >
> Hi Dmitri,
>
> We do tag. But I'm the CM guy who tags and there are 30 developers who
> commit/update more often than I tag. Unfortunately, their priority is
> higher than mine. :-(
>
> All, thanks for the quick answers. Just to clarify: our svn server is a
> linux box (Ubuntu 6.10, up-to-date, ext3 filesystem). Our developers
> aren't going to develop using linux because of lack of experience in
> this environment.
>
> --
>
>
> Joaquim Oliveira, MSc
>
>
> Analista de Sistemas
>
>
>
> Fone: +55 (85) 3216.7971
> Fax: +55 (85) 3216.7864
> Skype: joaquim.oliveira
>
>
> ISO 9001 : 2000 - CMMI3
>
>
>
>
> www.atlantico.com.br <http://www.atlantico.com.br>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: users-help@subversion.tigris.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Joaquim Oliveira <jo...@atlantico.com.br>.
Dmitri Colebatch escreveu:
> Do you guys tag at all? We've recently migrated a 16000 odd file
> repository and have certainly found that some operations are slower,
> but the tagging speed of svn compared to cvs is something that would
> definately stop me from switching back (o:
>
Hi Dmitri,
We do tag. But I'm the CM guy who tags and there are 30 developers who
commit/update more often than I tag. Unfortunately, their priority is
higher than mine. :-(
All, thanks for the quick answers. Just to clarify: our svn server is a
linux box (Ubuntu 6.10, up-to-date, ext3 filesystem). Our developers
aren't going to develop using linux because of lack of experience in
this environment.
--
Joaquim Oliveira, MSc
Analista de Sistemas
Fone: +55 (85) 3216.7971
Fax: +55 (85) 3216.7864
Skype: joaquim.oliveira
ISO 9001 : 2000 - CMMI3
www.atlantico.com.br <http://www.atlantico.com.br>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by "B. Smith-Mannschott" <be...@gmail.com>.
On Feb 7, 2007, at 15:33, Eric wrote:
> When doing an update or a commit, doesn't svn only send those files
> that have been changed?
True. However, this means groveling through the whole working copy
first to figure out *what* has changed. That, in turn means touching
a large number of small files in a large number of directories. This
kind of a task is hard for any file system. Windows' NTFS just
happens to be particularly bad at it.
// ben
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Eric <sp...@scoot.netis.com>.
At 09:24 AM 2/7/2007, Joaquim Oliveira wrote:
<JO>>>>>our team commit and update more than diff against the latest
version.<<<<<
Good morning, Joaquim.
Forgive the newbie question ... and I haven't yet used SVN on any
truly huge projects ... but...
When doing an update or a commit, doesn't svn only send those files
that have been changed?
Does each individual on your team really make such huge changes that
it takes so long to update or commit?
Or, are you saying that the sheer volume of files in the project
makes it take a long time for the client to step through all the .svn
files and project files to decide what to commit or update?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Joaquim Oliveira <jo...@atlantico.com.br>.
Adriano Ferreira escreveu:
> That's the price you pay (in disk size at least) for being able to
> make diffs of the working copy against the last version, without
> needing access to the server.
Adriano, I do know this. But it seems that using a filesystem like NTFS,
this approach has some limitations. As Andrew Bolstridge suggested in
another reply, maybe the admin files format could be redesigned. I don't
know if this is a common habit, but our team commit and update more than
diff against the latest version.
Thanks for the reply,
--
--
Joaquim Oliveira, MSc
Analista de Sistemas
Fone: +55 (85) 3216.7971
Fax: +55 (85) 3216.7864
Skype: joaquim.oliveira
ISO 9001 : 2000 - CMMI3
www.atlantico.com.br <http://www.atlantico.com.br>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Adriano Ferreira <a....@gmail.com>.
> On 2/7/07, Joaquim Oliveira <jo...@atlantico.com.br> wrote:
> > SVN access was made using svn:// protocol. CVS was faster in all
> > situations. Most of then, it is twice faster. For example, an update in
> > working copy root folder was about 20% faster in CVS.
> > We noticed that SVN creates more administrative files and directories
> > than CVS. The checkout size is:
> > - CVS: 24849 files, 4841 folders. Disk usage: 164 MB
> > - SVN: 27450 files, 22319 folders (!). Disk usage: 261 MB
That's the price you pay (in disk size at least) for being able to
make diffs of the working copy against the last version, without
needing access to the server.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Eric <sp...@scoot.netis.com>.
At 11:17 AM 2/7/2007, Steve Bakke wrote:
<SB>>>>>Updates are definitely not atomic as you could have an error
part-way through an update and your working copy will be in an
intermediate state. That's one of the reasons why you need 'svn cleanup'.<<<<<
Right, run svn cleanup followed by another svn update, right?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Steve Bakke <st...@amd.com>.
On 2/7/07 11:13 AM, "Eric" <sp...@scoot.netis.com> wrote:
>
> Question... commits are atomic, are "updates" also atomic?
Only changes to the repository are atomic. Updates are definitely not
atomic as you could have an error part-way through an update and your
working copy will be in an intermediate state. That's one of the reasons
why you need 'svn cleanup'.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Eric <sp...@scoot.netis.com>.
At 09:57 AM 2/7/2007, B. Smith-Mannschott wrote:
<BSM>>>>>Not fast. But, fast enough for my purposes. I certainly
wouldn't want to go back to CVS. Not only is rename nice, but the
efficient handling of binary files allows us to use SVN in situations
where we wouldn't have bothered to consider CVS.<<<<<
Also, if there are truly that many thousands and thousands of files,
I would think the atomic commit would be worth any conceivable level
of aggravation caused by lack of speed. I certainly wouldn't want to
go back to CVS on that basis alone.
Question... commits are atomic, are "updates" also atomic?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by "B. Smith-Mannschott" <be...@gmail.com>.
On Feb 7, 2007, at 13:02, Dmitri Colebatch wrote:
> Do you guys tag at all? We've recently migrated a 16000 odd file
> repository and have certainly found that some operations are slower,
> but the tagging speed of svn compared to cvs is something that would
> definately stop me from switching back (o:
Tagging sure is fast... we're forced by the way we chose to structure
our repository to use svncopy to pin our externals when we branch. This
entails
* checkout
* grovel, grovel, grovel, grovel
* commit
Not fast. But, fast enough for my purposes. I certainly wouldn't want
to go back to CVS. Not only is rename nice, but the efficient handling
of binary files allows us to use SVN in situations where we wouldn't
have bothered to consider CVS.
// ben
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Dmitri Colebatch <di...@colebatch.com>.
Do you guys tag at all? We've recently migrated a 16000 odd file
repository and have certainly found that some operations are slower,
but the tagging speed of svn compared to cvs is something that would
definately stop me from switching back (o:
On 2/7/07, Joaquim Oliveira <jo...@atlantico.com.br> wrote:
> Hi all,
>
> Our team tried svn for 4 months, but we're switching back to CVS due to
> performance problems. Our project is a huge java one: its structure has
> 10323 files and 2420 directories (117 MB). We made some tests using a
> variety of tools:
> - Eclipse 3.2 + Subclipse 1.1.9
> - Eclipse 3.2 + Subversive 1.0.0 rc4
> - Tortoise svn 1.4.1-7992
> - Eclipse 3.2 + internal CVS support
> - Tortoise CVS 1.8.30
>
> SVN access was made using svn:// protocol. CVS was faster in all
> situations. Most of then, it is twice faster. For example, an update in
> working copy root folder was about 20% faster in CVS.
> We noticed that SVN creates more administrative files and directories
> than CVS. The checkout size is:
> - CVS: 24849 files, 4841 folders. Disk usage: 164 MB
> - SVN: 27450 files, 22319 folders (!). Disk usage: 261 MB
>
> I searched the mail list archives, but couldn't find a solution for
> this. I found something about "the NTFS file system does not perform
> well when you have a large number of small files", but we need to
> develop in Windows, so adopting Linux/Ext3 is not an option. I've
> already seen these messages:
> - http://svn.haxx.se/users/archive-2005-04/1557.shtml
> -http://svn.haxx.se/users/archive-2005-04/1695.shtml
>
> We already tried disabling anti-virus software and upgrading to the
> latest version of the server (1.4.x) and plugins, but nothing worked.
> Developers complain a lot about this and, although SVN features are
> really better, a fast development environment is a must to our team.
>
> Is there any way to improve SVN performance? What are the most common
> bottlenecks?
>
> Thanks in advance,
>
> --
> --
>
>
> Joaquim Oliveira, MSc
>
>
> Analista de Sistemas
>
>
>
> Fone: +55 (85) 3216.7971
> Fax: +55 (85) 3216.7864
> Skype: joaquim.oliveira
>
>
> ISO 9001 : 2000 - CMMI3
>
>
>
>
> www.atlantico.com.br <http://www.atlantico.com.br>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: users-help@subversion.tigris.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Oliver Betz <li...@gmx.net>.
Johnathan Gifford wrote:
[...]
> seriously consider a Mac). We've found that turning off the fast
> indexing (as someone else mentioned) was a big help, but the killer is
Windows indexing is evil. Use a proper desktop search product and run
the indexing when you are out to lunch.
> anti-virus. With Subversion having two copies of every repository
on-access virus scanning is evil.
Are the developers wearing condoms all the day, too?
Antivirus software doesn't save responsible behaviour.
SCNR,
Oliver
--
Oliver Betz, Muenchen
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Les Mikesell <le...@gmail.com>.
Joaquim Oliveira wrote:
> I searched the mail list archives, but couldn't find a solution for
> this. I found something about "the NTFS file system does not perform
> well when you have a large number of small files", but we need to
> develop in Windows, so adopting Linux/Ext3 is not an option.
Other than needing a machine with some free software, running svn:,
ssh_svn: or https: protocols against a Linux server does nothing
problematic regarding developing on Windows. If you need to build or
test on the same physical machine you could run a copy of windows under
vmware.
--
Les Mikesell
lesmikesell@gmail.com
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Russ <rs...@istandfor.com>.
When we first started using svn, it was pretty slow as well... Then someone told me to disable windows indexing on the working copy and it made things much faster. For a small project like this, it would make a lot of sense to put it on its own partition, and use a caching utility (something that works like a ramdrive, but keeps things in sync with the hard disk). Do this, and although speeds probably wouldn't be faster than cvs on the same partition, they would be fast enough that you wouldn't notice.
Russ
Sent wirelessly via BlackBerry from T-Mobile.
-----Original Message-----
From: John Allen <jo...@dublinux.net>
Date: Wed, 07 Feb 2007 13:29:02
To:users@subversion.tigris.org
Subject: Re: Poor performance in windows. Switching back to CVS
Joaquim Oliveira wrote:
> Hi all,
>
> Our team tried svn for 4 months, but we're switching back to CVS due
> to performance problems. Our project is a huge java one: its structure
> has 10323 files and 2420 directories (117 MB). We made some tests
> using a variety of tools:
> - Eclipse 3.2 + Subclipse 1.1.9
> - Eclipse 3.2 + Subversive 1.0.0 rc4
> - Tortoise svn 1.4.1-7992
> - Eclipse 3.2 + internal CVS support
> - Tortoise CVS 1.8.30
>
> SVN access was made using svn:// protocol. CVS was faster in all
> situations. Most of then, it is twice faster. For example, an update
> in working copy root folder was about 20% faster in CVS.
Times please. Also when you are updating how many files have changed
between the revision
you have locally, and the head on the repository?
If you do and svn update, followed immediately by another svn update
does it still take ages?
Also I assume you are not checking out the entire repository, just the
trunk.
> We noticed that SVN creates more administrative files and directories
> than CVS. The checkout size is:
> - CVS: 24849 files, 4841 folders. Disk usage: 164 MB
> - SVN: 27450 files, 22319 folders (!). Disk usage: 261 MB
>
This is to be exepcted, as svn provides features that cvs does not, and
requires these extra
administrative files to do so.
> I searched the mail list archives, but couldn't find a solution for
> this. I found something about "the NTFS file system does not perform
> well when you have a large number of small files", but we need to
> develop in Windows, so adopting Linux/Ext3 is not an option. I've
> already seen these messages:
> - http://svn.haxx.se/users/archive-2005-04/1557.shtml
> -http://svn.haxx.se/users/archive-2005-04/1695.shtml
>
> We already tried disabling anti-virus software and upgrading to the
> latest version of the server (1.4.x) and plugins, but nothing worked.
> Developers complain a lot about this and, although SVN features are
> really better, a fast development environment is a must to our team.
>
> Is there any way to improve SVN performance? What are the most common
> bottlenecks?
>
> Thanks in advance,
>
--
John Allen mailto:john.allen@codemountain.net
CodeMountain http://www.codemountain.net
Ubuntu 6.10, kernel 2.6.17-10-generic
up 12 days, 21:21, 16 users, load average: 0.37, 0.20, 0.29
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
RE: Re: Poor performance in windows. Switching back to CVS
Posted by "Bolstridge, Andrew" <an...@intergraph.com>.
> > We noticed that SVN creates more administrative files and
> directories
> > than CVS. The checkout size is:
> > - CVS: 24849 files, 4841 folders. Disk usage: 164 MB
> > - SVN: 27450 files, 22319 folders (!). Disk usage: 261 MB
> >
> This is to be exepcted, as svn provides features that cvs
> does not, and requires these extra administrative files to do so.
I don't think the issue is that extra admin files exist, but the
quantity of them. If svn used a single xml file for all the working copy
admin details instead of a .svn file (yes, I know I'm being very
over-simplistic), then there wouldn't be an issue.
Mind you, a single admin file per versioned file would mean SVN could
start to operate on individual files, not directories... That'd be a
good thing.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by John Allen <jo...@dublinux.net>.
Joaquim Oliveira wrote:
> Hi all,
>
> Our team tried svn for 4 months, but we're switching back to CVS due
> to performance problems. Our project is a huge java one: its structure
> has 10323 files and 2420 directories (117 MB). We made some tests
> using a variety of tools:
> - Eclipse 3.2 + Subclipse 1.1.9
> - Eclipse 3.2 + Subversive 1.0.0 rc4
> - Tortoise svn 1.4.1-7992
> - Eclipse 3.2 + internal CVS support
> - Tortoise CVS 1.8.30
>
> SVN access was made using svn:// protocol. CVS was faster in all
> situations. Most of then, it is twice faster. For example, an update
> in working copy root folder was about 20% faster in CVS.
Times please. Also when you are updating how many files have changed
between the revision
you have locally, and the head on the repository?
If you do and svn update, followed immediately by another svn update
does it still take ages?
Also I assume you are not checking out the entire repository, just the
trunk.
> We noticed that SVN creates more administrative files and directories
> than CVS. The checkout size is:
> - CVS: 24849 files, 4841 folders. Disk usage: 164 MB
> - SVN: 27450 files, 22319 folders (!). Disk usage: 261 MB
>
This is to be exepcted, as svn provides features that cvs does not, and
requires these extra
administrative files to do so.
> I searched the mail list archives, but couldn't find a solution for
> this. I found something about "the NTFS file system does not perform
> well when you have a large number of small files", but we need to
> develop in Windows, so adopting Linux/Ext3 is not an option. I've
> already seen these messages:
> - http://svn.haxx.se/users/archive-2005-04/1557.shtml
> -http://svn.haxx.se/users/archive-2005-04/1695.shtml
>
> We already tried disabling anti-virus software and upgrading to the
> latest version of the server (1.4.x) and plugins, but nothing worked.
> Developers complain a lot about this and, although SVN features are
> really better, a fast development environment is a must to our team.
>
> Is there any way to improve SVN performance? What are the most common
> bottlenecks?
>
> Thanks in advance,
>
--
John Allen mailto:john.allen@codemountain.net
CodeMountain http://www.codemountain.net
Ubuntu 6.10, kernel 2.6.17-10-generic
up 12 days, 21:21, 16 users, load average: 0.37, 0.20, 0.29
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Johnathan Gifford <jg...@wernervas.com>.
We have a ColdFusion project in our repository that has a 3 to 1 ratio of files to directories (bad design, but no need to go there). With about 2000+ directories, the Mac OS X developers quickly execute checkouts and all other functions. The Windows developers literally crawl on checkout and all other functions (enough to make some of them seriously consider a Mac). We've found that turning off the fast indexing (as someone else mentioned) was a big help, but the killer is anti-virus. With Subversion having two copies of every repository file in the working copy that is constantly changing, anti-virus sucks up the processor(s) and IO really fast checking all of those files. We have notice that Norton can be configured to be less aggressive than Mcaffe can. But a good configuration of the anti-virus is the biggest win that can happen.
Maybe it's time that the Subversion stewards and committers take this up with the anti-virus manufacturers and figure out a way to improve this without sacrificing the safety of Windows based PC's.
Johnathan
>>> On Tue, Feb 6, 2007 at 1:40 PM, in message
<45...@atlantico.com.br>, Joaquim Oliveira
<jo...@atlantico.com.br> wrote:
> Hi all,
>
> Our team tried svn for 4 months, but we're switching back to CVS due to
> performance problems. Our project is a huge java one: its structure has
> 10323 files and 2420 directories (117 MB). We made some tests using a
> variety of tools:
> - Eclipse 3.2 + Subclipse 1.1.9
> - Eclipse 3.2 + Subversive 1.0.0 rc4
> - Tortoise svn 1.4.1- 7992
> - Eclipse 3.2 + internal CVS support
> - Tortoise CVS 1.8.30
>
> SVN access was made using svn:// protocol. CVS was faster in all
> situations. Most of then, it is twice faster. For example, an update in
> working copy root folder was about 20% faster in CVS.
> We noticed that SVN creates more administrative files and directories
> than CVS. The checkout size is:
> - CVS: 24849 files, 4841 folders. Disk usage: 164 MB
> - SVN: 27450 files, 22319 folders (!). Disk usage: 261 MB
>
> I searched the mail list archives, but couldn't find a solution for
> this. I found something about "the NTFS file system does not perform
> well when you have a large number of small files", but we need to
> develop in Windows, so adopting Linux/Ext3 is not an option. I've
> already seen these messages:
> - http://svn.haxx.se/users/archive- 2005- 04/1557.shtml
> - http://svn.haxx.se/users/archive- 2005- 04/1695.shtml
>
> We already tried disabling anti- virus software and upgrading to the
> latest version of the server (1.4.x) and plugins, but nothing worked.
> Developers complain a lot about this and, although SVN features are
> really better, a fast development environment is a must to our team.
>
> Is there any way to improve SVN performance? What are the most common
> bottlenecks?
>
> Thanks in advance,
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Marcus Rohrmoser <mr...@gmx-gmbh.de>.
Talden schrieb:
> On 2/13/07, Marcus Rohrmoser <mr...@gmx-gmbh.de> wrote:
>> Hi Joaquim,
>>
>> Joaquim Oliveira schrieb:
>> > 10323 files and 2420 directories (117 MB). We made some tests using a
>>
>> that's quite a lot for a single project.
>
> Heh. If that's quite a lot we have bigger problems.
I considered that as big from my point of view only. There sure are bigger projects that svn handles
quite well - especially compared to CVS and over slow networks. There were some threads in the past
about pushing svn to it's limits with lots of files and/or huge files.
> PS: At which point does ext3 begin to suffer for having too many
> revision files in the FSFS back-end?
I didn't hear of any advantage of BDB over FSFS here recently. There was a discussion about running
out of inodes some time ago. Thought it was here, but a quick search got me
http://svn.haxx.se/dev/archive-2005-10/0264.shtml.
I'd start with FSFS and switch to BDB later if necessary.
Greetings,
M
Re: Poor performance in windows. Switching back to CVS
Posted by Talden <ta...@gmail.com>.
On 2/13/07, Marcus Rohrmoser <mr...@gmx-gmbh.de> wrote:
> Hi Joaquim,
>
> Joaquim Oliveira schrieb:
> > 10323 files and 2420 directories (117 MB). We made some tests using a
>
> that's quit a lot for a single project. I think as long as you don't
> change to a quicker filesystem or split the project into handier pieces
> svn won't make you happy.
Heh. If that's quite a lot we have bigger problems. from CVS an export
(not the working copy which is much worse) is some 17,000 files, 3,500
files and a touch over 400MB. As a working copy in CVS this is twice
the number of files, in Subversion this must be much worse with
additional book-keeping and pristine copies (I've yet to have a
cvs2svn migration complete for me to asses this on a full working
copy).
We definitely need to break up our project but currently this is the
build-test model in place. I hope that, once I have a file CVS to SVN
migration complete that I am not disappointed.
> Bye the way - how long takes a typical (commandline) svn update? svn status?
For us, the CVS update is a terrible measure. We're on the opposite
hemisphere to our CVS server over what the New Zealand communications
industry laughingly calls broadband. CVS is dog-slow, so much so that
even with the file-system performance issues of the large file/folder
counts, network IO is much much worse.
I hope subversion is better rather than worse on this large working copy.
PS: At which point does ext3 begin to suffer for having too many
revision files in the FSFS back-end? IE an update or a checkout has
to find these by name - surely even in ext3 finding the appropriate
file is a function of the number of files. I ask this because initial
smaller, more manageable cvs2svn converts suggest a full conversion
will yield upwards of 60,000+ revisions - should I be considering BDB?
--
Talden
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
RE: Re: Poor performance in windows. Switching back to CVS
Posted by Rob van Oostrum <ro...@blastradius.com>.
One thing I've seen done is to give each developer their own NFS share
on a Linux/Unix box, either the same machine that runs your SVN server
or something more dedicated. That way, you can develop on Windows by
mounting your share as a drive, and you can leverage the Unix/Linux
performance advantages of the SVN client by keeping an SSH session open
for that purpose.
R.
> -----Original Message-----
> From: Marcus Rohrmoser [mailto:mrohrmoser@gmx-gmbh.de]
> Sent: Monday, February 12, 2007 3:18 PM
> To: users@subversion.tigris.org
> Cc: Joaquim Oliveira
> Subject: Re: Poor performance in windows. Switching back to CVS
>
> Hi Joaquim,
>
> Joaquim Oliveira schrieb:
> > 10323 files and 2420 directories (117 MB). We made some tests using
a
>
> that's quit a lot for a single project. I think as long as you don't
> change to a quicker filesystem or split the project into handier
pieces
> svn won't make you happy.
>
> Bye the way - how long takes a typical (commandline) svn update? svn
> status?
>
> Greetings,
> M
>
> P.S.: Just to be curious, how long takes Eclipse without svn to start
> and to refresh?
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: users-help@subversion.tigris.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Talden <ta...@gmail.com>.
Knowing the filename doesn't mean you know where it is on the disk.
'find' from a file-system point of view means resolving a
path+filename to a location on disk. This degrades as the number of
paths in the file-system increases. The mechanism used to store the
lookup tables for these files affects performance and degrades in
different ways depending on the technique and sometimes the pattern of
the paths themselves (some file-systems are very fast at resolving the
filename given a folder node, others lump all path+file names into a
single storage structure that is fast to search but slow to update.
NTFS, I've heard, is known to suffer with very large numbers of files
per folder. I have no idea of the performance degradation pattern of
the EXT file-system.
--
Talden
On 2/14/07, Jeff Smith <js...@robotronics.com> wrote:
> On Monday 12 February 2007 15:01, Talden wrote:
> > PS: At which point does ext3 begin to suffer for having too many
> > revision files in the FSFS back-end? IE an update or a checkout
> > has to find these by name - surely even in ext3 finding the
> > appropriate file is a function of the number of files. I ask this
> > because initial smaller, more manageable cvs2svn converts suggest a
> > full conversion will yield upwards of 60,000+ revisions - should I
> > be considering BDB?
> >
> > --
> > Talden
>
> Interresting question... what do you mean by "find", does it not
> already know the path?
>
> I think that so long as the file system itself handles that many files
> (for example not running out of inodes), then the only thing is more
> files take longer to read and process. That should be expected, and
> no version control system would be unaffected by the increase.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: users-help@subversion.tigris.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Jeff Smith <js...@robotronics.com>.
On Monday 12 February 2007 15:01, Talden wrote:
> PS: At which point does ext3 begin to suffer for having too many
> revision files in the FSFS back-end? IE an update or a checkout
> has to find these by name - surely even in ext3 finding the
> appropriate file is a function of the number of files. I ask this
> because initial smaller, more manageable cvs2svn converts suggest a
> full conversion will yield upwards of 60,000+ revisions - should I
> be considering BDB?
>
> --
> Talden
Interresting question... what do you mean by "find", does it not
already know the path?
I think that so long as the file system itself handles that many files
(for example not running out of inodes), then the only thing is more
files take longer to read and process. That should be expected, and
no version control system would be unaffected by the increase.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Poor performance in windows. Switching back to CVS
Posted by Marcus Rohrmoser <mr...@gmx-gmbh.de>.
Hi Joaquim,
Joaquim Oliveira schrieb:
> 10323 files and 2420 directories (117 MB). We made some tests using a
that's quit a lot for a single project. I think as long as you don't
change to a quicker filesystem or split the project into handier pieces
svn won't make you happy.
Bye the way - how long takes a typical (commandline) svn update? svn status?
Greetings,
M
P.S.: Just to be curious, how long takes Eclipse without svn to start
and to refresh?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org