You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by "Harvey, Edward" <Ed...@patni.com> on 2008/04/05 23:28:07 UTC
performance enhancement by working copy svn server
Basically, just wondering if anyone has ever thought along these lines ...
I was thinking about how to speed up operations, such as svn status, etc, which are currently required to walk the tree. With large numbers of files, this can be expensive. The root cause is simple - peoples files can be changed by any process without the knowledge of svn; therefore svn must inspect each file to see if it's changed. To improve this would require some process to have the ability to constantly monitor all file access within the working copy.
So the idea is - Suppose there were a special "working copy" svn server, which could have exclusive access to some folder on the system, containing the working copy, and then export those files to localhost, where they could be mounted into a user's home directory (or wherever). The user would work in his/her home directory as always, but the "working copy svn server" would be able to monitor all file access. No change of behavior would be necessary; the working copy could contain versioned and unversioned files exactly as it always has, but the "working copy svn server" could keep track of all the local modifications on the fly, and eliminate the need to ever walk the tree.
Granted, root would have to assist somehow, to let the filesystem mount. And I'm sure it's not a trivial code to write.
This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, e-transmission dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error kindly delete this e-mail from your records. If it appears that this mail has been forwarded to you without proper authority, please notify us immediately at netadmin@patni.com and delete this mail.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by Eric Gillespie <ep...@pretzelnet.org>.
Listman <li...@burble.net> writes:
> can you give us a little more insight here? how else are you going to
> avoid
> scanning the entire tree for changes without an svn edit command?
It will have to scan the entire tree when not in edit mode.
--
Eric Gillespie <*> epg@pretzelnet.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by Listman <li...@burble.net>.
On Apr 11, 2008, at 9:03 PM- Apr 11, 2008, Eric Gillespie wrote:
> "David Glasser" <gl...@davidglasser.net> writes:
>
>> By the way, a nice thing about this model that epg pointed out
>> recently is that this mode meshes well with optional text-base-free
>> operation. You can get away without storing a separate text-base for
>> the read-only files; 'svn edit' would add a copy of the text-base to
>> the metadata area before making the file writable.
>
> In fact, this is the *only* way that svn edit makes sense to me.
>
> Don't worry, folks, svn edit will not be required in wc-ng.
can you give us a little more insight here? how else are you going to
avoid
scanning the entire tree for changes without an svn edit command?
i'm happy to type in one extra command if it means i can see what files
have changed in a second rather than several minutes.
will the new approach have that kind of performance (using the repos
example i quoted earlier in this thread).
thx
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by Eric Gillespie <ep...@pretzelnet.org>.
"David Glasser" <gl...@davidglasser.net> writes:
> By the way, a nice thing about this model that epg pointed out
> recently is that this mode meshes well with optional text-base-free
> operation. You can get away without storing a separate text-base for
> the read-only files; 'svn edit' would add a copy of the text-base to
> the metadata area before making the file writable.
In fact, this is the *only* way that svn edit makes sense to me.
Don't worry, folks, svn edit will not be required in wc-ng.
--
Eric Gillespie <*> epg@pretzelnet.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by David Glasser <gl...@davidglasser.net>.
On Thu, Apr 10, 2008 at 8:22 AM, Ben Collins-Sussman
<su...@red-bean.com> wrote:
> On Thu, Apr 10, 2008 at 10:01 AM, Harvey, Edward
> <Ed...@patni.com> wrote:
> > > libsvn_wc rewrite which
> > > (1) centralizes metadata in one place and
> > > (2) allows users the option of 'svn edit' to tell svn what you're
> >
> > > changing, so that 'svn status/update/commit' no longer need
> > > to traverse the tree.
> >
> > With the wc rewrite, are you talking about somehow preventing extraneous processes, such as "vi somefile" from accessing files in a WC "behind the back" of the svn client?
>
> Just like perforce, the whole working copy would have read-only
> permissions set by default; 'svn edit' would make a file read-write.
>
> Yes, this system is imperfect, but it works pretty well in practice in
> conjunction with user education. We already have this behavior when
> the 'svn:needs-lock' property is attached to the file... we'd just be
> generalizing the behavior to cover the whole working copy. (Some
> naughty tools ignore read-only permissions, but typcially not text
> editors.)
By the way, a nice thing about this model that epg pointed out
recently is that this mode meshes well with optional text-base-free
operation. You can get away without storing a separate text-base for
the read-only files; 'svn edit' would add a copy of the text-base to
the metadata area before making the file writable.
--dave
--
David Glasser | glasser@davidglasser.net | http://www.davidglasser.net/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
RE: performance enhancement by working copy svn server
Posted by Garance A Drosihn <dr...@rpi.edu>.
At 5:16 PM -0400 4/12/08, Garance A Drosihn wrote:
>At 3:00 PM -0400 4/10/08, Harvey, Edward wrote:
>> > The whole working copy would have
>>> read-only permissions set by default; 'svn edit' would make
>>> a file read-write.
>>
>>Actually, I think this sounds very reasonable (as an option; I do
>>respect the need for people to continue working the way they always
>>have before). It's a quick & dirty, easy simple technique to
>>nearly eliminate the whole problem. No architecture dependent
>>(well, minimal) code, which still (usually) creates the ability for
>>the svn client to monitor all the file edits.
>
>Let me add this observation: You might also need to have directories
>to be marked as read-only by default. There are some commands which
>will alter a file by making changes to a copy of it, and them removing
>the old file by moving in the new file. I've had .svn working copies
>corrupted by this, for instance, even though the *files* in the .svn
>directory are permitted read-only to me.
I better add a bit more to this comment...
I realize that making the directories read-only by default may cause
some headaches for normal use of a working copy. But my point is
that unless you do that, it isn't really safe to assume that the
files in the working copy will not be changed just because we've
added this 'svn edit' trickery that is based solely on chmod bits.
--
Garance Alistair Drosehn = gad@gilead.netel.rpi.edu
Senior Systems Programmer or gad@freebsd.org
Rensselaer Polytechnic Institute or drosih@rpi.edu
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
RE: performance enhancement by working copy svn server
Posted by Garance A Drosihn <dr...@rpi.edu>.
At 3:00 PM -0400 4/10/08, Harvey, Edward wrote:
> > The whole working copy would have
>> read-only permissions set by default; 'svn edit' would make
>> a file read-write.
>
>Actually, I think this sounds very reasonable (as an option; I do
>respect the need for people to continue working the way they always
>have before). It's a quick & dirty, easy simple technique to nearly
>eliminate the whole problem. No architecture dependent (well,
>minimal) code, which still (usually) creates the ability for the svn
>client to monitor all the file edits.
Let me add this observation: You might also need to have directories
to be marked as read-only by default. There are some commands which
will alter a file by making changes to a copy of it, and them removing
the old file by moving in the new file. I've had .svn working copies
corrupted by this, for instance, even though the *files* in the .svn
directory are permitted read-only to me.
--
Garance Alistair Drosehn = gad@gilead.netel.rpi.edu
Senior Systems Programmer or gad@freebsd.org
Rensselaer Polytechnic Institute or drosih@rpi.edu
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by Mattias Engdegård <ma...@virtutech.se>.
John Peacock <jo...@havurah-software.org> writes:
>I think that you underestimate the sheer scale of the performance drag caused by
>the existing WC code that will evaporate with centralized meta data *alone*.
Very true. For a rough estimate of the cost of the current WC
organisation, benchmark checkout/update/commit on an ordinary
(persistent) file system against tmpfs, where most operations are
extremely cheap[1]. A good WC design would attempt to minimise the
differences by using file system operations that are fast in both.
>The other thing that you are missing is that you (and I) are geeks, comfortable
>with a wide variety of sometimes cryptic commandline tools.
There are also rather sophisticated users who want their patch and
merge tools to work without having to be hacked to know about 'svn edit',
not to mention the other ad-hoc scripts and shell command lines we run
to change 4711 source files at the same time.
---
[1] So much cheaper, in fact, that I frequently use tmpfs for
working copies despite the slight danger of losing recent
changes in a kernel panic (which never happens anyway).
With a better WC organisation, this wouldn't be necessary.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by John Peacock <jo...@havurah-software.org>.
Ben Collins-Sussman wrote:
> I have to admit, when I first started using perforce a couple of years
> ago, it seemed strangely restrictive compared to cvs or svn. But now
> running 'p4 edit' doesn't seem any weirder to me than running 'svn
> add', 'svn rm', 'svn mv', or 'svn cp'. It's just one more case of
> telling the system what you're doing. And holy awesome, what an
> incredible speed boost you get in return!
You forget that I use SVK; I already get the speed boost and I don't have to
declare I am editing a file. ;-)
I think that you underestimate the sheer scale of the performance drag caused by
the existing WC code that will evaporate with centralized meta data *alone*.
Until proven otherwise, I think the whole issue of 'svn edit' should be
considered orthoganal to centralizing the meta data. If the WC-NG design
_requires_ requesting the edit of a file, then I think it needs to be rewritten.
There is already a model for centralized metadata that doesn't have that
limitation (see above). I'm not saying the feature isn't valuable (and may even
be required) in certain specific circumstances. And for some people, it might
make a very big difference in their workflow. But it should not be anything
more than another optional mode of operation.
The other thing that you are missing is that you (and I) are geeks, comfortable
with a wide variety of sometimes cryptic commandline tools. For every one of
our ilk, there are a thousand people who either don't know about version control
at all, but could benefit from it immeasurably, or use it and think it is
something magical. They don't use the commandline tools at all. They use
software we would never consider (WordPad anyone?) to change files and wouldn't
have the slightest interest in running some other command (or clicking some
other requester in Windows Explorer) before they change a file. There are also
the many tools which will happily flip the ReadOnly flag (just to be helpful),
and blow away your carefully constructed WC.
John
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by Stefan Sperling <st...@elego.de>.
On Thu, Apr 10, 2008 at 07:55:45PM -0700, Peter Wemm wrote:
> On the 3-way merge stuff.. In perforce, the format is like this:
>
> >>> original
> orig
> === theirs
> their version
> --- yours
> your version
> <<<
>
> In svn, by default, you don't get to see the original text (just like cvs):
> <<<<<<< .mine
> your version
> =======
> their version
> >>>>>>> .r2
>
> With a 1-line change, you can sort-of get the p4-like semantics:
> <<<<<<< .mine
> your version
> ||||||| .r1
> orig
> =======
> their change
> >>>>>>> .r2
>
> libsvn_wc/merge.c, in the call for svn_diff_file_output_merge():
> - FALSE, /* display original */
> + TRUE, /* display original */
Wheee! Shouldn't we make this a new configurable option in 1.6?
--
Stefan Sperling <st...@elego.de> Software Monkey
German law requires the following banner :(
elego Software Solutions GmbH HRB 77719
Gustav-Meyer-Allee 25, Gebaeude 12 Tel: +49 30 23 45 86 96
13355 Berlin Fax: +49 30 23 45 86 95
http://www.elego.de CEO: Olaf Wagner
Store password unencrypted (yes/no)? No
Re: performance enhancement by working copy svn server
Posted by Peter Wemm <pe...@wemm.org>.
On Thu, Apr 10, 2008 at 6:10 PM, Ben Collins-Sussman
<su...@red-bean.com> wrote:
> On Thu, Apr 10, 2008 at 3:43 PM, John Peacock
> <jo...@havurah-software.org> wrote:
> > Benjamin Smith-Mannschott wrote:
> >
> > > Yuck. At first blush, this seems to me a pretty unnatural way to work.
> > >
> >
> > Me too.
>
> I think hundreds of thousands of perforce users would disagree with
> you guys. ;-)
>
> I have to admit, when I first started using perforce a couple of years
> ago, it seemed strangely restrictive compared to cvs or svn. But now
> running 'p4 edit' doesn't seem any weirder to me than running 'svn
> add', 'svn rm', 'svn mv', or 'svn cp'. It's just one more case of
> telling the system what you're doing. And holy awesome, what an
> incredible speed boost you get in return!
I've used perforce since before it was called perforce.. (p3.com,
remember that? I've been using it for something like 11 or 12 years
now). I have a constant love-hate relationship with it. There are
many things that they got right, but.. by god, did they screw up in
some colossal ways. If the WC-NG conversation is going to include p4
comparisons in it, then I'd love to vent some steam about what they
got wrong (and right).
The single biggest hassle with perforce, is that they went much
further with metadata centralization than we're talking about here..
The client-side metadata is actually stored on the server. This is
fine for small groups, or even large groups where there's an
administrator with a big stick and the authority to use it (eg: in a
company). We tried this on freebsd.org and it just doesn't scale. We
have a huge tree, and our developers love to check things out, all
over the place. Before we knew it, the server database was sinking in
client side metadata. (It's looking like we're going to need to go to
32GB of ram on the server. svn on a similar sized tree seems to be
comfortable in 8GB)
Having to do a 'p4 edit' is something that you get used to. It's
really the least of the problems.
The other big problem is that the shell client is antiquated even by
1996's standards. It has *no* directory awareness. If you put stray
files in there, it can't tell you. There is no 'p4 status' command or
equivalent. All p4 can tell you, is what files you told the server
that you were editing. People who use p4 for non-trivial things end
up writing scripts that scan for stray files and so on.
IMHO, while having p4 tell you when a file is open for edit by another
user is kinda neat, it really isn't worth the cost. If your
development model is branch centric (like it is with p4 on
freebsd.org), it is very rare that another person will be also editing
a file at the same time (well, besides you, on another machine).
The horrific cost of storing metadata on the server, the
administrative overhead and the antiquated command line client are the
killer reasons why p4 was a non starter for consideration for
freebsd.org.
On the other hand, p4 got a few things very right. Some of which I'll
sorely miss.
* client views. Your working copy is 'mapped' into server space via a
flexible table. You can synthesize a checked out tree from bits
scattered all over the server repository.
* branch views. Branches are tables of 'copy here to there, then
here2 to there2, then here3 to there3, etc'. A "branch" in p4 speak
is a name for this src,dest copy list. The advantage of this is that
populating the branch is atomic. With svn you'd have to do multiple
copies, remember the change numbers, etc.
* integrated configuration for things like file type mappings.
Instead of lame things like putting mime mappings in ~/.svn/config,
the pathname to file type mappings are stored on the p4 server (p4
typemap). So instead of having to beat people over the head to
remember to set 'binary' or 'keywords' on some file type, you can just
set shared mime-like mappings on the server. (The client can, of
course, override this. but the server can set the defaults)
* pervasive merge tracking as a first class feature. (think merginfo,
except 12 years ago)
* sane revision numbering, both for global changes and file revs.
Files are numbered 1,2,3,... in the tree as well as global change set
numbers. 6+ digit file revision numbers suck. Our security officer
folks are having a fit about how to specify version numbers in
advisories.
* saner client-side 3-way merging when resolving conflicts. (see
footnote). svn can do this, but requires a trivial code change.
(this could easily be in the config file)
* server specified commit template text. Even CVS supports this.
I honestly expect that most of our serious devlopers on freebsd.org
will use svk or git-svn for local work, but the svn client is mostly
good enough. I've already discovered that we have to put together an
extensive set of hooks to enforce settings for mime types, binary,
eol, keywords, etc since we can't depend on the clients getting it
right by default. (And to deal with custom keywords.. sigh.)
On the 3-way merge stuff.. In perforce, the format is like this:
>>> original
orig
=== theirs
their version
--- yours
your version
<<<
In svn, by default, you don't get to see the original text (just like cvs):
<<<<<<< .mine
your version
=======
their version
>>>>>>> .r2
With a 1-line change, you can sort-of get the p4-like semantics:
<<<<<<< .mine
your version
||||||| .r1
orig
=======
their change
>>>>>>> .r2
libsvn_wc/merge.c, in the call for svn_diff_file_output_merge():
- FALSE, /* display original */
+ TRUE, /* display original */
-Peter
--
Peter Wemm - peter@wemm.org; peter@FreeBSD.org; peter@yahoo-inc.com
"All of this is for nothing if we don't go to the stars" - JMS/B5
"If Java had true garbage collection, most programs would delete
themselves upon execution." -- Robert Sewell
**WANTED TO BUY: Garmin Streetpilot 2650 or 2660. Not later model! **
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by "Ph. Marek" <ph...@bmlv.gv.at>.
On Dienstag, 15. April 2008, Harvey, Edward wrote:
> > I agree wholeheartedly with what John wrote in his reply. Scanning a
> > disk can be done a lot faster than we're currently doing; centralized
> > metadata helps, but there are obviously other tricks, too -- just as an
> > example, look at how quickly Google Picasa can scan your whole disk for
> > images!
>
> I challenge the idea that scanning the whole tree could possibly ever be
> "fast."
If seek time can be eliminated, by using a central data storage and going
through the inodes in "correct" order, you can save *much* time.
Ever tried FSVS? I'll quote my previous email; for 22147 (known) entries (ie.
that exist in the repository and are just checked for changes) on *cold*
caches:
# time fsvs st > /dev/null
real 0m6.000s
user 0m0.240s
sys 0m0.372s
Not bad, no?
That's on Linux, ext3.
> Yes, if you do it once and repeat it, then you're benefitting from
> the cache on the 2nd run - For my users that makes the difference between
> 10 minutes and 30 seconds. But they're only going to svn up (or whatever)
> ... once, twice, three times a day ...
How often do you do "svn st"?
> And the system is doing actual work
> in parallel. So they never benefit from the cache. Every time they scan
> the tree, it takes 10 minutes.
Well, if you always purge your OS's cache, you'll have to go to the harddisk
to get the information ...
> I considered seriously, writing a cron script to tar all the .svn dirs into
> /dev/null, and stat the whole working copy of every user, ever 5 minutes.
> Just to keep that info in cache. But I finally decided that was stupid
> enough to abandon it.
tar is "intelligent" enough to not bother with the data if it sees that it
writes to /dev/null - you'd have to use /dev/zero instead.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by "Martin v. Löwis" <Ma...@hpi.uni-potsdam.de>.
> I challenge the idea that scanning the whole tree could possibly ever
> be "fast." Yes, if you do it once and repeat it, then you're
> benefitting from the cache on the 2nd run - For my users that makes
> the difference between 10 minutes and 30 seconds. But they're only
> going to svn up (or whatever) ... once, twice, three times a day ...
> And the system is doing actual work in parallel. So they never
> benefit from the cache. Every time they scan the tree, it takes 10
> minutes.
For how many files? On what file system?
10 minutes is definitely too long. You can scan the disk much much much
faster. A Python script I just wrote that stat's /usr on Linux
(partially cached), I can stat 323347 files in 213s.
Regards,
Martin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by Greg Hudson <gh...@MIT.EDU>.
On Sat, 2008-04-19 at 02:12 -0400, Daniel Berlin wrote:
> >
> > (note: I'm not suggesting this as a specific solution in any way...)
> >
> We already do exactly this :)
>
> (cache stat info in the WC and compare it)
Actually, not quite. There's a way of eliminating
svn_sleep_for_timestamps() in there, at the cost of doing file content
comparisons if the timestamp of .svn/entries (the "index") matches the
timestamp of the file.
To answer eg's mail, no, nothing in there would make the tree scan any
faster than it already is. If a tree-walk and lstat() of every file is
too slow to be tolerable for "svn st", then the three discussed
solutions (svn edit, file server for the working copy, relying on arcane
system-dependent interfaces to track file edits) are the only ways to
achieve acceptable performance.
(A tree-walk and lstat() of every file is probably fast enough on any
size project with a decent filesystem, but not everyone works under
decent filesystems.)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by Daniel Berlin <db...@dberlin.org>.
>
> (note: I'm not suggesting this as a specific solution in any way...)
>
We already do exactly this :)
(cache stat info in the WC and compare it)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by eg <eg...@gmail.com>.
David Glasser wrote:
> On Wed, Apr 16, 2008 at 10:16 PM, Benjamin Smith-Mannschott
> <bs...@gmail.com> wrote:
>> It seems to me that Mercurial to compensate for the performance issues
>> that SVN's working copy format so successfully demonstrates.
>
> Yes, the revlog format is designed to deal with the performance issues
> of stitching together a bunch of deltas with as few seeks as possible.
> This is similar to issues that Subversion faces on the server.
>
> It has nothing to do with efficiently locating changed files in the
> user's working tree.
>
ah, right... with all the talk of svn server, and centralized metadata
improvements, file seeks, etc.. I guess I was barking up the wrong tree.
Sorry for the noise.
Probably something like the following is more relevant to see what
machinations other vcs's go through to try and attain efficiency
(without resorting to "svn edit" type modes):
http://www.kernel.org/pub/software/scm/git/docs/technical/racy-git.txt
(note: I'm not suggesting this as a specific solution in any way...)
I am hoping that the wc-ng improvements are such that this "svn edit"
option isn't really necessary to use except for very large scale cases
(wishful?)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by David Glasser <gl...@davidglasser.net>.
On Wed, Apr 16, 2008 at 10:16 PM, Benjamin Smith-Mannschott
<bs...@gmail.com> wrote:
>
> On Apr 16, 2008, at 19:12, David Glasser wrote:
> ... snip...
>
> > > I'm convinced that a disk scan can be made orders of magnitude
> > > faster than ours is now, even without FS-specific tricks (like,
> > > e.g., walking the $MFT on NTFS)
> > >
> > > Agreed.
> > > Isn't Mercurial, with their revlog format, an example of
> > > an implementation which demonstrates this ability to be much
> > > faster for these type of operations.
> >
> > Totally unrelated.
>
> How so?
>
> This representation seems to be designed *specifically* to account for
> the essentially sequential nature of disk access. (i.e. random access
> is not cheap.)
>
> http://www.selenic.com/mercurial/wiki/index.cgi/Revlog
> > With one read of the index to fetch the record and then one read
> > of the revlog, Mercurial can reconstruct any version of a file
> > in time proportional to the file size.
> ...snip...
> > So that adding a new version requires only O(1) seeks, the
> > revlogs and their indices are append-only.
>
> http://hgbook.red-bean.com/hgbookch4.html#x8-870004.5.4
> > 4.5.4 Avoiding seeks
> >
> > Critical to Mercurial's performance is the avoidance of seeks of the
> > disk head, since any seek is far more expensive than even a
> > comparatively large read operation.
> ...snip...
>
> It seems to me that Mercurial to compensate for the performance issues
> that SVN's working copy format so successfully demonstrates.
Yes, the revlog format is designed to deal with the performance issues
of stitching together a bunch of deltas with as few seeks as possible.
This is similar to issues that Subversion faces on the server.
It has nothing to do with efficiently locating changed files in the
user's working tree.
--dave
--
David Glasser | glasser@davidglasser.net | http://www.davidglasser.net/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by Benjamin Smith-Mannschott <bs...@gmail.com>.
On Apr 16, 2008, at 19:12, David Glasser wrote:
... snip...
> > I'm convinced that a disk scan can be made orders of magnitude
> > faster than ours is now, even without FS-specific tricks (like,
> > e.g., walking the $MFT on NTFS)
> >
> > Agreed.
> > Isn't Mercurial, with their revlog format, an example of
> > an implementation which demonstrates this ability to be much
> > faster for these type of operations.
>
> Totally unrelated.
How so?
This representation seems to be designed *specifically* to account for
the essentially sequential nature of disk access. (i.e. random access
is not cheap.)
http://www.selenic.com/mercurial/wiki/index.cgi/Revlog
> With one read of the index to fetch the record and then one read
> of the revlog, Mercurial can reconstruct any version of a file
> in time proportional to the file size.
...snip...
> So that adding a new version requires only O(1) seeks, the
> revlogs and their indices are append-only.
http://hgbook.red-bean.com/hgbookch4.html#x8-870004.5.4
> 4.5.4 Avoiding seeks
>
> Critical to Mercurial’s performance is the avoidance of seeks of the
> disk head, since any seek is far more expensive than even a
> comparatively large read operation.
...snip...
It seems to me that Mercurial to compensate for the performance issues
that SVN's working copy format so successfully demonstrates.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by David Glasser <gl...@davidglasser.net>.
On Tue, Apr 15, 2008 at 2:59 PM, eg <eg...@gmail.com> wrote:
> Branko Čibej wrote:
>
> >
> > >
> > > If you're already turning everything upside-down, and willing to trade
> > > memory for performance, it might be worth thinking about an optional
> > > svn-client-daemon that uses
> > > inotify/FSEvents/FindFirstChangeNotification/etc. Then you'd only have
> > > to do a full scan once per boot; the OS would tell you about any
> > > further modifications.
> > >
> > >
> >
> > Ouch. This introduces a long-living daemon that all other bits depend on,
> that has to be stable and never make a mistake. Not a good way to design
> robust software, IMHO. This falls under the "unnecessary complexity" and
> "premature optimization" headings. I'm convinced that a disk scan can be
> made orders of magnitude faster than ours is now, even without FS-specific
> tricks (like, e.g., walking the $MFT on NTFS)
> >
> >
>
> Agreed.
> Isn't Mercurial, with their revlog format, an example of an implementation
> which demonstrates this ability to be much faster for these type of
> operations.
Totally unrelated.
--dave
--
David Glasser | glasser@davidglasser.net | http://www.davidglasser.net/
Re: performance enhancement by working copy svn server
Posted by eg <eg...@gmail.com>.
Branko Čibej wrote:
>>
>> If you're already turning everything upside-down, and willing to trade
>> memory for performance, it might be worth thinking about an optional
>> svn-client-daemon that uses
>> inotify/FSEvents/FindFirstChangeNotification/etc. Then you'd only have
>> to do a full scan once per boot; the OS would tell you about any
>> further modifications.
>>
>
> Ouch. This introduces a long-living daemon that all other bits depend
> on, that has to be stable and never make a mistake. Not a good way to
> design robust software, IMHO. This falls under the "unnecessary
> complexity" and "premature optimization" headings. I'm convinced that a
> disk scan can be made orders of magnitude faster than ours is now, even
> without FS-specific tricks (like, e.g., walking the $MFT on NTFS)
>
Agreed.
Isn't Mercurial, with their revlog format, an example of an
implementation which demonstrates this ability to be much faster for
these type of operations.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by Branko Čibej <br...@xbc.nu>.
Peter Harris wrote:
> On Tue, Apr 15, 2008 at 11:00 AM, Branko Čibej wrote:
>
>> Harvey, Edward wrote:
>>
>>> I challenge the idea that scanning the whole tree could possibly ever be "fast."
>>>
>> There is vast room for improvement here, but it implies a radical change in
>> WC design; not only centralizing metadata, but also turning the whole
>> scanning concept upside down. We'll have to give control of the scan to the
>> code that knows about the underlying organization (i.e., stop doing it in
>> editor drives), and likely have to stop trying to be streamy (i.e., trade
>> memory footprint for performance).
>>
>
> If you're already turning everything upside-down, and willing to trade
> memory for performance, it might be worth thinking about an optional
> svn-client-daemon that uses
> inotify/FSEvents/FindFirstChangeNotification/etc. Then you'd only have
> to do a full scan once per boot; the OS would tell you about any
> further modifications.
>
Ouch. This introduces a long-living daemon that all other bits depend
on, that has to be stable and never make a mistake. Not a good way to
design robust software, IMHO. This falls under the "unnecessary
complexity" and "premature optimization" headings. I'm convinced that a
disk scan can be made orders of magnitude faster than ours is now, even
without FS-specific tricks (like, e.g., walking the $MFT on NTFS)
> Given that all OSs seem to have something inotify-like, and they are
> all slightly different, this seems like the sort of thing that APR
> should cover. My APR-fu is weak, however. I can't find anything like
> this in the APR docs.
>
It's not there.
-- Brane
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by Peter Harris <sv...@peter.is-a-geek.org>.
On Tue, Apr 15, 2008 at 11:00 AM, Branko Čibej wrote:
> Harvey, Edward wrote:
> >
> > I challenge the idea that scanning the whole tree could possibly ever be "fast."
>
> There is vast room for improvement here, but it implies a radical change in
> WC design; not only centralizing metadata, but also turning the whole
> scanning concept upside down. We'll have to give control of the scan to the
> code that knows about the underlying organization (i.e., stop doing it in
> editor drives), and likely have to stop trying to be streamy (i.e., trade
> memory footprint for performance).
If you're already turning everything upside-down, and willing to trade
memory for performance, it might be worth thinking about an optional
svn-client-daemon that uses
inotify/FSEvents/FindFirstChangeNotification/etc. Then you'd only have
to do a full scan once per boot; the OS would tell you about any
further modifications.
Given that all OSs seem to have something inotify-like, and they are
all slightly different, this seems like the sort of thing that APR
should cover. My APR-fu is weak, however. I can't find anything like
this in the APR docs.
Peter Harris
Re: performance enhancement by working copy svn server
Posted by Branko Čibej <br...@xbc.nu>.
Harvey, Edward wrote:
>> I agree wholeheartedly with what John wrote in his reply. Scanning a
>> disk can be done a lot faster than we're currently doing; centralized
>> metadata helps, but there are obviously other tricks, too -- just as an
>> example, look at how quickly Google Picasa can scan your whole disk for
>> images!
>>
>
> I challenge the idea that scanning the whole tree could possibly ever be "fast."
Ignoring caching of filesystem metadata, for an initial estimate,
performance is bound by the number of seeks and non-sequential accesses.
Our current WC scanning code excels in both categories, because of the
explosion of small files that have to be read, and the out-of-order
directory scanning.
There is vast room for improvement here, but it implies a radical change
in WC design; not only centralizing metadata, but also turning the whole
scanning concept upside down. We'll have to give control of the scan to
the code that knows about the underlying organization (i.e., stop doing
it in editor drives), and likely have to stop trying to be streamy
(i.e., trade memory footprint for performance).
Just recently I came across an interesting example of how naively
written code can seriously degrade performance of a disk-based
application; I happened to connect a large NTFS USB disk full of videos
to a Mac. Both Windows and Mac OS are too smart for their own good and
aggressively scan new volumes. For that particular drive, it took
Windows about a minute or so before it magnanimously allowed me to start
using the drive. I gave up on the Mac after 10 minutes. ...
-- Brane
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
RE: performance enhancement by working copy svn server
Posted by "Harvey, Edward" <Ed...@patni.com>.
> I agree wholeheartedly with what John wrote in his reply. Scanning a
> disk can be done a lot faster than we're currently doing; centralized
> metadata helps, but there are obviously other tricks, too -- just as an
> example, look at how quickly Google Picasa can scan your whole disk for
> images!
I challenge the idea that scanning the whole tree could possibly ever be "fast." Yes, if you do it once and repeat it, then you're benefitting from the cache on the 2nd run - For my users that makes the difference between 10 minutes and 30 seconds. But they're only going to svn up (or whatever) ... once, twice, three times a day ... And the system is doing actual work in parallel. So they never benefit from the cache. Every time they scan the tree, it takes 10 minutes.
I considered seriously, writing a cron script to tar all the .svn dirs into /dev/null, and stat the whole working copy of every user, ever 5 minutes. Just to keep that info in cache. But I finally decided that was stupid enough to abandon it.
This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, e-transmission dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error kindly delete this e-mail from your records. If it appears that this mail has been forwarded to you without proper authority, please notify us immediately at netadmin@patni.com and delete this mail.
Re: performance enhancement by working copy svn server
Posted by Karl Fogel <kf...@red-bean.com>.
Branko Čibej <br...@xbc.nu> writes:
> Making "svn edit" a required operation is asking for trouble, because
> it's too easy to work around the requirement. The only viable option
> would be to use FS change notifications on systems that support it,
> but since there aren't too many of those, we shouldn't rely on such
> things to get acceptable performance.
There's a lot of FUD flying around in this thread.
(Understandably, since it's a big topic and few people have time to
follow it in the detail that would be required not to fall victim to
FUD... The Subversion dev@ list somewhat resembles the U.S. presidental
campaign in that respect, I guess.)
Anyway: no one is talking about making 'svn edit' a required operation.
No one has ever proposed tat, as far as I know. It would be optional.
Whether it would someday be the default or not, I don't know.
As for it being too easy to work around it (when the client
configuration is set up to "require" 'svn edit' in the first place):
The experience of other popular VC systems that already have this
requirement answers that concern. When there's real-world data out
there, we are relieved of the burden of speculation!
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by Branko Čibej <br...@xbc.nu>.
Ben Collins-Sussman wrote:
> On Thu, Apr 10, 2008 at 3:43 PM, John Peacock
> <jo...@havurah-software.org> wrote:
>
>> Benjamin Smith-Mannschott wrote:
>>
>>
>>> Yuck. At first blush, this seems to me a pretty unnatural way to work.
>>>
>>>
>> Me too.
>>
>
> I think hundreds of thousands of perforce users would disagree with
> you guys. ;-)
>
Feh. Hundreds of thousands of CVS users thought CVS was great until they
tried SVN.
> I have to admit, when I first started using perforce a couple of years
> ago, it seemed strangely restrictive compared to cvs or svn. But now
> running 'p4 edit' doesn't seem any weirder to me than running 'svn
> add', 'svn rm', 'svn mv', or 'svn cp'. It's just one more case of
> telling the system what you're doing. And holy awesome, what an
> incredible speed boost you get in return!
>
I agree wholeheartedly with what John wrote in his reply. Scanning a
disk can be done a lot faster than we're currently doing; centralized
metadata helps, but there are obviously other tricks, too -- just as an
example, look at how quickly Google Picasa can scan your whole disk for
images!
Making "svn edit" a required operation is asking for trouble, because
it's too easy to work around the requirement. The only viable option
would be to use FS change notifications on systems that support it, but
since there aren't too many of those, we shouldn't rely on such things
to get acceptable performance.
-- Brane
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by Branko Čibej <br...@xbc.nu>.
Stefan Sperling wrote:
> On Sat, May 17, 2008 at 04:10:53PM -0400, Harvey, Edward wrote:
>
>> <anecdote>
>> For me in particular, this will be very welcome. I have a badly-behaved tool (from Cadence) and this is its behavior:
>> The Cadence tool assumes it owns a whole directory that you're working on.
>> It goes in, modifies all the stuff it cares to modify...
>> It finds the ".svn" directory, deletes it, and creates a ".svn" file containing only a Space char.
>> And then I'm unhappy.
>> </anecdote>
>>
>
> What a crazy tool.
>
Unfortunately those seem to be quite common on the Mac. We've had issues
open about that since before 1.0.
-- Brane
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by Stefan Sperling <st...@elego.de>.
On Sat, May 17, 2008 at 04:10:53PM -0400, Harvey, Edward wrote:
> <anecdote>
> For me in particular, this will be very welcome. I have a badly-behaved tool (from Cadence) and this is its behavior:
> The Cadence tool assumes it owns a whole directory that you're working on.
> It goes in, modifies all the stuff it cares to modify...
> It finds the ".svn" directory, deletes it, and creates a ".svn" file containing only a Space char.
> And then I'm unhappy.
> </anecdote>
What a crazy tool.
I have another badly-behaved tool on my system that cannot work
recursively with Subversion working copies in a nice way,
unless I filter its output.
It's called '/usr/bin/grep' :)
Stefan
Re: performance enhancement by working copy svn server
Posted by Listman <li...@burble.net>.
On May 17, 2008, at 2:28 PM- May 17, 2008, Steinar Bang wrote:
>>>>>> Listman <li...@burble.net>:
>
>> Agreed, this is one of the advantages of Perforce, the meta-data is
>> saved on the server not in the WC and there's no chance of a
>> misbehaving tool breaking the working copy..
>
> Storing the WC on the server, is a behaviour of Perforce I would not
> like subversion to replicate. It causes problems with scaling on the
> server, and limits/complicates offline operations.
>
> (not that I think that storing the WC on the server is the plan. The
> way I've read the discussions on the list, the "single file or subdir"
> mentioned above, will be on the client machine)
Subversion's working copy gets broken regularly in our environment and
other companies I've worked. Scattering it across the filesystem is a
recipe for disaster in my experience.
The reality is that 90% of companies will put their user accounts (and
hence the Subversion meta-data) on a fileserver. Its seldom that
the .svn directories actually local to the client machine.
This idea of "offline operations" maybe useful in small, open-source
type activities, but isn't the reality in most corporations.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by Karl Fogel <kf...@red-bean.com>.
Steinar Bang <sb...@dod.no> writes:
> (not that I think that storing the WC on the server is the plan. The
> way I've read the discussions on the list, the "single file or subdir"
> mentioned above, will be on the client machine)
That is correct.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by Steinar Bang <sb...@dod.no>.
>>>>> Listman <li...@burble.net>:
> Agreed, this is one of the advantages of Perforce, the meta-data is
> saved on the server not in the WC and there's no chance of a
> misbehaving tool breaking the working copy..
Storing the WC on the server, is a behaviour of Perforce I would not
like subversion to replicate. It causes problems with scaling on the
server, and limits/complicates offline operations.
(not that I think that storing the WC on the server is the plan. The
way I've read the discussions on the list, the "single file or subdir"
mentioned above, will be on the client machine)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by Listman <li...@burble.net>.
On May 17, 2008, at 1:10 PM- May 17, 2008, Harvey, Edward wrote:
>>> I think the general consensus is that the default svn behavior will
>>> not change; 'svn edit' will be an optional mode of working,
>> for those
>>> who want or need the speed tradeoff.
>>
>> I figured that from later postings in the thread, but it's
>> good to see it spelled out clear.
>
> Correct, the default behavior will not change - 'svn edit' remains
> optional.
>
> However, the WC backend is likely to change. Consolidating all
> the .svn subdirs into a single file, or into a single dir.
>
> <anecdote>
> For me in particular, this will be very welcome. I have a badly-
> behaved tool (from Cadence) and this is its behavior:
> The Cadence tool assumes it owns a whole directory that
> you're working on.
> It goes in, modifies all the stuff it cares to modify...
> It finds the ".svn" directory, deletes it, and creates a
> ".svn" file containing only a Space char.
> And then I'm unhappy.
> </anecdote>
Agreed, this is one of the advantages of Perforce, the meta-data is
saved on the server not in the WC and there's no chance of a
misbehaving tool breaking the working copy.. As one of my users said
the other day, Perforce is "bullet-proof", Subversion is fragile..
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
RE: performance enhancement by working copy svn server
Posted by "Harvey, Edward" <Ed...@patni.com>.
> > I think the general consensus is that the default svn behavior will
> > not change; 'svn edit' will be an optional mode of working,
> for those
> > who want or need the speed tradeoff.
>
> I figured that from later postings in the thread, but it's
> good to see it spelled out clear.
Correct, the default behavior will not change - 'svn edit' remains optional.
However, the WC backend is likely to change. Consolidating all the .svn subdirs into a single file, or into a single dir.
<anecdote>
For me in particular, this will be very welcome. I have a badly-behaved tool (from Cadence) and this is its behavior:
The Cadence tool assumes it owns a whole directory that you're working on.
It goes in, modifies all the stuff it cares to modify...
It finds the ".svn" directory, deletes it, and creates a ".svn" file containing only a Space char.
And then I'm unhappy.
</anecdote>
This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, e-transmission dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error kindly delete this e-mail from your records. If it appears that this mail has been forwarded to you without proper authority, please notify us immediately at netadmin@patni.com and delete this mail.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by Steinar Bang <sb...@dod.no>.
>>>>> "Ben Collins-Sussman" <su...@red-bean.com>:
> I think the general consensus is that the default svn behavior will
> not change; 'svn edit' will be an optional mode of working, for those
> who want or need the speed tradeoff.
I figured that from later postings in the thread, but it's good to see
it spelled out clear.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by Ben Collins-Sussman <su...@red-bean.com>.
On Fri, May 16, 2008 at 3:36 AM, Steinar Bang <sb...@dod.no> wrote:
>>>>>> "Ben Collins-Sussman" <su...@red-bean.com>:
>
>> On Thu, Apr 10, 2008 at 3:43 PM, John Peacock
>> <jo...@havurah-software.org> wrote:
>>> Benjamin Smith-Mannschott wrote:
>
>>>> Yuck. At first blush, this seems to me a pretty unnatural way to work.
>
>>> Me too.
>
>> I think hundreds of thousands of perforce users would disagree with
>> you guys. ;-)
>
> As a counter example, I am a Perforce user who still thinks it's an
> unnatural way to work. I would not like to see this the default
> behaviour of svn.
I think the general consensus is that the default svn behavior will
not change; 'svn edit' will be an optional mode of working, for those
who want or need the speed tradeoff.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by Steinar Bang <sb...@dod.no>.
>>>>> "Ben Collins-Sussman" <su...@red-bean.com>:
> On Thu, Apr 10, 2008 at 3:43 PM, John Peacock
> <jo...@havurah-software.org> wrote:
>> Benjamin Smith-Mannschott wrote:
>>> Yuck. At first blush, this seems to me a pretty unnatural way to work.
>> Me too.
> I think hundreds of thousands of perforce users would disagree with
> you guys. ;-)
As a counter example, I am a Perforce user who still thinks it's an
unnatural way to work. I would not like to see this the default
behaviour of svn.
(But then I've never gotten really friendly with Perforce, and in my list
of Perforce annoyances, it's a fairly minor one...)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by Ben Collins-Sussman <su...@red-bean.com>.
On Thu, Apr 10, 2008 at 3:43 PM, John Peacock
<jo...@havurah-software.org> wrote:
> Benjamin Smith-Mannschott wrote:
>
> > Yuck. At first blush, this seems to me a pretty unnatural way to work.
> >
>
> Me too.
I think hundreds of thousands of perforce users would disagree with
you guys. ;-)
I have to admit, when I first started using perforce a couple of years
ago, it seemed strangely restrictive compared to cvs or svn. But now
running 'p4 edit' doesn't seem any weirder to me than running 'svn
add', 'svn rm', 'svn mv', or 'svn cp'. It's just one more case of
telling the system what you're doing. And holy awesome, what an
incredible speed boost you get in return!
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by John Peacock <jo...@havurah-software.org>.
Benjamin Smith-Mannschott wrote:
> Yuck. At first blush, this seems to me a pretty unnatural way to work.
Me too.
> I'd much rather see this addressed at the file system level either by
> copy-on-write or something like (apple's) fseventsd. I realize that
> this is (way) out of scope for subversion, which needs a solution that
> will work on any conventional file system. Pitty.
This seems way too complicated also (and likely nigh-unportable).
Just as an FYI, SVK has centralized metadata (in the form of an ordinary
Subversion repository actually) and it really doesn't seem to take all
that long to scan the working copy to locate the changed files (I belive
that SVK also caches the changed files as it sees them, so it is faster
the second time round, say 'svk st' followed by 'svk diff'). But then
again I am not using WC's with [tens of] thousands of files either.
John
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by Benjamin Smith-Mannschott <bs...@gmail.com>.
On Apr 10, 2008, at 21:00, Harvey, Edward wrote:
>> The whole working copy would have
>> read-only permissions set by default; 'svn edit' would make
>> a file read-write.
>
> Actually, I think this sounds very reasonable (as an option; I do
> respect the need for people to continue working the way they always
> have before). It's a quick & dirty, easy simple technique to nearly
> eliminate the whole problem. No architecture dependent (well,
> minimal) code, which still (usually) creates the ability for the svn
> client to monitor all the file edits.
Yuck. At first blush, this seems to me a pretty unnatural way to work.
I'd much rather see this addressed at the file system level either by
copy-on-write or something like (apple's) fseventsd. I realize that
this is (way) out of scope for subversion, which needs a solution that
will work on any conventional file system. Pitty.
// ben
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
RE: performance enhancement by working copy svn server
Posted by "Harvey, Edward" <Ed...@patni.com>.
> The whole working copy would have
> read-only permissions set by default; 'svn edit' would make
> a file read-write.
Actually, I think this sounds very reasonable (as an option; I do respect the need for people to continue working the way they always have before). It's a quick & dirty, easy simple technique to nearly eliminate the whole problem. No architecture dependent (well, minimal) code, which still (usually) creates the ability for the svn client to monitor all the file edits.
Would this be a feature in 1.5, or after 1.5? Cuz I'd definitely have an immediate need for it.
This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, e-transmission dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error kindly delete this e-mail from your records. If it appears that this mail has been forwarded to you without proper authority, please notify us immediately at netadmin@patni.com and delete this mail.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by Ben Collins-Sussman <su...@red-bean.com>.
On Thu, Apr 10, 2008 at 10:01 AM, Harvey, Edward
<Ed...@patni.com> wrote:
> > libsvn_wc rewrite which
> > (1) centralizes metadata in one place and
> > (2) allows users the option of 'svn edit' to tell svn what you're
>
> > changing, so that 'svn status/update/commit' no longer need
> > to traverse the tree.
>
> With the wc rewrite, are you talking about somehow preventing extraneous processes, such as "vi somefile" from accessing files in a WC "behind the back" of the svn client?
Just like perforce, the whole working copy would have read-only
permissions set by default; 'svn edit' would make a file read-write.
Yes, this system is imperfect, but it works pretty well in practice in
conjunction with user education. We already have this behavior when
the 'svn:needs-lock' property is attached to the file... we'd just be
generalizing the behavior to cover the whole working copy. (Some
naughty tools ignore read-only permissions, but typcially not text
editors.)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by "Ph. Marek" <ph...@bmlv.gv.at>.
On Donnerstag, 10. April 2008, Ben Collins-Sussman wrote:
> Try using subversion
> with a 700MB source tree; running 'svn status' takes minutes to crawl
> the tree (along with commits and updates).
>
> > And having the metadata localized allows to do wierd things like
> > manually merge WC temporarily, or easily take a smaller piece of the WC
> > and move it elsewhere.
>
> That's what we've been telling ourselves for 8 years; it turns out
> that nobody actually ever breaks working copies into small portable
> pieces. And the price we pay for scattered metadata is huge.
I just want to chime in with some hard data for the interested: FSVS uses a
single file for most of the needed data (all that's needed for "status"), and
it's bit faster.
See here for an example:
http://fsvs.tigris.org/source/browse/fsvs/trunk/fsvs/doc/PERFORMANCE?rev=972&view=markup
This is when a single file (sorted by inode-number) is read, the entries
lstat()ed in inode order, and no keyword substitution is possible (so that a
size change is immediately seen as changed data).
Regards,
Phil
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by "Martin v. Löwis" <Ma...@hpi.uni-potsdam.de>.
> I'd agree with this. I can't understand how anyone would see "adequate" svn
> performance when the repository gets large (read 10's of 1000's of files).
>
> Note that "warming the cache" 90% of the time doesn't help - my company
> has ~ 50
> users and the cache quickly gets out of date..
I don't understand why the number of users would matter anything.
Are these users all using the same workstation also? Stat caching
is a per-workstation thing.
Reegards,
Martin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by David Glasser <gl...@davidglasser.net>.
On Fri, Apr 11, 2008 at 3:15 PM, Listman <li...@burble.net> wrote:
>
>
> On Apr 11, 2008, at 3:05 PM- Apr 11, 2008, David Glasser wrote:
>
>
> > On Fri, Apr 11, 2008 at 2:56 PM, Listman <li...@burble.net> wrote:
> >
> > >
> > >
> > >
> > >
> > > >
> > > > On Thu, Apr 10, 2008 at 3:43 PM, John Peacock
> > > > <jo...@havurah-software.org> wrote:
> > > >
> > > > > Benjamin Smith-Mannschott wrote:
> > > > >
> > > > >
> > > > > > Yuck. At first blush, this seems to me a pretty unnatural way to
> work.
> > > > > >
> > > > > >
> > > > >
> > > > > Me too.
> > > > >
> > > > I think hundreds of thousands of perforce users would disagree with
> > > > you guys. ;-)
> > > >
> > > >
> > > > I have to admit, when I first started using perforce a couple of years
> > > > ago, it seemed strangely restrictive compared to cvs or svn. But now
> > > > running 'p4 edit' doesn't seem any weirder to me than running 'svn
> > > > add', 'svn rm', 'svn mv', or 'svn cp'. It's just one more case of
> > > > telling the system what you're doing. And holy awesome, what an
> > > > incredible speed boost you get in return!
> > > >
> > > >
> > > >
> > >
> > > I'd agree with this. I can't understand how anyone would see "adequate"
> svn
> > > performance when the repository gets large (read 10's of 1000's of
> files).
> > >
> > > Note that "warming the cache" 90% of the time doesn't help - my company
> has
> > > ~ 50
> > > users and the cache quickly gets out of date..
> > >
> > > we also have perforce here and i thought you'd be interested in seeing a
> > > comparison.
> > >
> > >
> > > for a 505MB working copy with 36,781 objects
> > >
> > > comparing Subversion and Perforce:
> > >
> > > SUBVERSION
> > > [cad@x09 /home/cad/trials/svn/work] -57- % time svn status
> > > ? fab7
> > > 1.614u 5.141s 5:09.28 2.1% 0+0k 0+0io 8pf+0w
> > >
> > >
> > > PERFORCE
> > > [cad@x09 /home/cad/trials/versic/p4/work] -78- % time p4 opened ...
> > > ... - file(s) not opened on this client.
> > > 0.000u 0.000s 0:00.00 0.0% 0+0k 0+0io 0pf+0w
> > >
> > > [sbutler@x07 /home/sbutler/trials/p4/work] -70- % time p4 sync
> > > File(s) up-to-date.
> > > 0.000u 0.000s 0:00.14 0.0% 0+0k 0+0io 0pf+0w
> > >
> > >
> > > so subversion took 5:09min to report back and p4 less than a second.
> > >
> > >
> > > NOTE that svn update is very slow also.
> > >
> >
> > Perforce and Subversion have different scaling characteristics. Some
> > situations work better for Perforce; some work better for Subversion;
> > others would work better for Subversion if the wc was improved.
> >
> >
>
> for a "typical" company with say 100 users on a fast local network with
> user accounts on a netapp fileserver, what are the scaling benefits of
> Subversion?
At this point I think we've moved from dev@ topics to users@ topics.
(My svn performance goals are aimed at repositories with quite a bit
more than 100 users.)
--dave
--
David Glasser | glasser@davidglasser.net | http://www.davidglasser.net/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by Listman <li...@burble.net>.
On Apr 11, 2008, at 3:05 PM- Apr 11, 2008, David Glasser wrote:
> On Fri, Apr 11, 2008 at 2:56 PM, Listman <li...@burble.net> wrote:
>>
>>
>>
>>>
>>> On Thu, Apr 10, 2008 at 3:43 PM, John Peacock
>>> <jo...@havurah-software.org> wrote:
>>>> Benjamin Smith-Mannschott wrote:
>>>>
>>>>> Yuck. At first blush, this seems to me a pretty unnatural way to
>>>>> work.
>>>>>
>>>>
>>>> Me too.
>>> I think hundreds of thousands of perforce users would disagree with
>>> you guys. ;-)
>>>
>>>
>>> I have to admit, when I first started using perforce a couple of
>>> years
>>> ago, it seemed strangely restrictive compared to cvs or svn. But
>>> now
>>> running 'p4 edit' doesn't seem any weirder to me than running 'svn
>>> add', 'svn rm', 'svn mv', or 'svn cp'. It's just one more case of
>>> telling the system what you're doing. And holy awesome, what an
>>> incredible speed boost you get in return!
>>>
>>>
>>
>> I'd agree with this. I can't understand how anyone would see
>> "adequate" svn
>> performance when the repository gets large (read 10's of 1000's of
>> files).
>>
>> Note that "warming the cache" 90% of the time doesn't help - my
>> company has
>> ~ 50
>> users and the cache quickly gets out of date..
>>
>> we also have perforce here and i thought you'd be interested in
>> seeing a
>> comparison.
>>
>>
>> for a 505MB working copy with 36,781 objects
>>
>> comparing Subversion and Perforce:
>>
>> SUBVERSION
>> [cad@x09 /home/cad/trials/svn/work] -57- % time svn status
>> ? fab7
>> 1.614u 5.141s 5:09.28 2.1% 0+0k 0+0io 8pf+0w
>>
>>
>> PERFORCE
>> [cad@x09 /home/cad/trials/versic/p4/work] -78- % time p4 opened ...
>> ... - file(s) not opened on this client.
>> 0.000u 0.000s 0:00.00 0.0% 0+0k 0+0io 0pf+0w
>>
>> [sbutler@x07 /home/sbutler/trials/p4/work] -70- % time p4 sync
>> File(s) up-to-date.
>> 0.000u 0.000s 0:00.14 0.0% 0+0k 0+0io 0pf+0w
>>
>>
>> so subversion took 5:09min to report back and p4 less than a second.
>>
>>
>> NOTE that svn update is very slow also.
>
> Perforce and Subversion have different scaling characteristics. Some
> situations work better for Perforce; some work better for Subversion;
> others would work better for Subversion if the wc was improved.
>
for a "typical" company with say 100 users on a fast local network with
user accounts on a netapp fileserver, what are the scaling benefits of
Subversion?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by David Glasser <gl...@davidglasser.net>.
On Fri, Apr 11, 2008 at 2:56 PM, Listman <li...@burble.net> wrote:
>
>
>
> >
> > On Thu, Apr 10, 2008 at 3:43 PM, John Peacock
> > <jo...@havurah-software.org> wrote:
> > > Benjamin Smith-Mannschott wrote:
> > >
> > > > Yuck. At first blush, this seems to me a pretty unnatural way to work.
> > > >
> > >
> > > Me too.
> > I think hundreds of thousands of perforce users would disagree with
> > you guys. ;-)
> >
> >
> > I have to admit, when I first started using perforce a couple of years
> > ago, it seemed strangely restrictive compared to cvs or svn. But now
> > running 'p4 edit' doesn't seem any weirder to me than running 'svn
> > add', 'svn rm', 'svn mv', or 'svn cp'. It's just one more case of
> > telling the system what you're doing. And holy awesome, what an
> > incredible speed boost you get in return!
> >
> >
>
> I'd agree with this. I can't understand how anyone would see "adequate" svn
> performance when the repository gets large (read 10's of 1000's of files).
>
> Note that "warming the cache" 90% of the time doesn't help - my company has
> ~ 50
> users and the cache quickly gets out of date..
>
> we also have perforce here and i thought you'd be interested in seeing a
> comparison.
>
>
> for a 505MB working copy with 36,781 objects
>
> comparing Subversion and Perforce:
>
> SUBVERSION
> [cad@x09 /home/cad/trials/svn/work] -57- % time svn status
> ? fab7
> 1.614u 5.141s 5:09.28 2.1% 0+0k 0+0io 8pf+0w
>
>
> PERFORCE
> [cad@x09 /home/cad/trials/versic/p4/work] -78- % time p4 opened ...
> ... - file(s) not opened on this client.
> 0.000u 0.000s 0:00.00 0.0% 0+0k 0+0io 0pf+0w
>
> [sbutler@x07 /home/sbutler/trials/p4/work] -70- % time p4 sync
> File(s) up-to-date.
> 0.000u 0.000s 0:00.14 0.0% 0+0k 0+0io 0pf+0w
>
>
> so subversion took 5:09min to report back and p4 less than a second.
>
>
> NOTE that svn update is very slow also.
Perforce and Subversion have different scaling characteristics. Some
situations work better for Perforce; some work better for Subversion;
others would work better for Subversion if the wc was improved.
--dave
--
David Glasser | glasser@davidglasser.net | http://www.davidglasser.net/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by Listman <li...@burble.net>.
> On Thu, Apr 10, 2008 at 3:43 PM, John Peacock
> <jo...@havurah-software.org> wrote:
> > Benjamin Smith-Mannschott wrote:
> >
> > > Yuck. At first blush, this seems to me a pretty unnatural way to
> work.
> > >
> >
> > Me too.
> I think hundreds of thousands of perforce users would disagree with
> you guys. ;-)
>
> I have to admit, when I first started using perforce a couple of years
> ago, it seemed strangely restrictive compared to cvs or svn. But now
> running 'p4 edit' doesn't seem any weirder to me than running 'svn
> add', 'svn rm', 'svn mv', or 'svn cp'. It's just one more case of
> telling the system what you're doing. And holy awesome, what an
> incredible speed boost you get in return!
>
I'd agree with this. I can't understand how anyone would see
"adequate" svn
performance when the repository gets large (read 10's of 1000's of
files).
Note that "warming the cache" 90% of the time doesn't help - my
company has ~ 50
users and the cache quickly gets out of date..
we also have perforce here and i thought you'd be interested in seeing
a comparison.
for a 505MB working copy with 36,781 objects
comparing Subversion and Perforce:
SUBVERSION
[cad@x09 /home/cad/trials/svn/work] -57- % time svn status
? fab7
1.614u 5.141s 5:09.28 2.1% 0+0k 0+0io 8pf+0w
PERFORCE
[cad@x09 /home/cad/trials/versic/p4/work] -78- % time p4 opened ...
... - file(s) not opened on this client.
0.000u 0.000s 0:00.00 0.0% 0+0k 0+0io 0pf+0w
[sbutler@x07 /home/sbutler/trials/p4/work] -70- % time p4 sync
File(s) up-to-date.
0.000u 0.000s 0:00.14 0.0% 0+0k 0+0io 0pf+0w
so subversion took 5:09min to report back and p4 less than a second.
NOTE that svn update is very slow also.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by Karl Fogel <kf...@red-bean.com>.
"Oren Eini (Murphy & Associates)" <v-...@microsoft.com> writes:
> From my perspective, I really like SVN's ability to figure what
> changed on its own, without having to be told. And having the
> metadata localized allows to do wierd things like manually merge WC
> temporarily, or easily take a smaller piece of the WC and move it
> elsewhere.
The ability to figure out what has changed won't go away, but it would
be nice to give uses a way to choose which way they want to swing on the
speed-vs-convenience tradeoff.
Moving smaller pieces of the WC elsewhere is a feature that will be
preserved, via an 'svn detach' command (I'm making up the name, but you
get the idea).
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by David Glasser <gl...@davidglasser.net>.
On Thu, Apr 10, 2008 at 11:50 AM, Peter Wemm <pe...@wemm.org> wrote:
> On Wed, Apr 9, 2008 at 3:41 PM, Oren Eini (Murphy & Associates)
>
> <v-...@microsoft.com> wrote:
>
> > Um, do I count as someone who actually breaks working copies?
> > That is not a highly used feature, I'll admit, and not one that I'll particulary miss.
> >
> > Thanks for clarifying on its optionality.
>
> It's also something we use in freebsd.org fairly extensively in cvs.
> svn's ability to cope with that sort abuse was one of the things in
> its favour when we were looking at svn vs git vs hg. (We're in the
> process of doing a trial conversion right now before we commit. We've
> still got some huge problems with svn to solve.)
>
> The main things we do are partial subtree checkouts (eg: check out
> src/sys or or src/usr.bin/ls or all of src) and moving subtrees around
> (eg: mv src/sys src/sys.wip). We also have people who merge
> independent checkouts together. eg: checkout -d foo src/sys; edit
> foo; mv src/sys /src/sys.clean; mv foo src/sys.
>
> Nobody from our camp would mind having to issue commands to inform svn
> of the evil that we've just done behind its back, just so long as it
> is possible somehow to do it. Be it commands in the CLI, or scripts
> using API calls to massage the metadata, or whatever.
IMHO, when we put in the effort to write a new centralized workspace
library, supporting situations like "working on a bunch of disjoint
parts of repositories at the same time" should in fact be a prime
motivator. (As far as I'm concerned the theoretical ability to do
this is a major advantage of svn over git/hg/etc, but the brittleness
of the wc layer makes Subversion not live up to its potential.)
--dave
--
David Glasser | glasser@davidglasser.net | http://www.davidglasser.net/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by Peter Wemm <pe...@wemm.org>.
On Wed, Apr 9, 2008 at 3:41 PM, Oren Eini (Murphy & Associates)
<v-...@microsoft.com> wrote:
> Um, do I count as someone who actually breaks working copies?
> That is not a highly used feature, I'll admit, and not one that I'll particulary miss.
>
> Thanks for clarifying on its optionality.
It's also something we use in freebsd.org fairly extensively in cvs.
svn's ability to cope with that sort abuse was one of the things in
its favour when we were looking at svn vs git vs hg. (We're in the
process of doing a trial conversion right now before we commit. We've
still got some huge problems with svn to solve.)
The main things we do are partial subtree checkouts (eg: check out
src/sys or or src/usr.bin/ls or all of src) and moving subtrees around
(eg: mv src/sys src/sys.wip). We also have people who merge
independent checkouts together. eg: checkout -d foo src/sys; edit
foo; mv src/sys /src/sys.clean; mv foo src/sys.
Nobody from our camp would mind having to issue commands to inform svn
of the evil that we've just done behind its back, just so long as it
is possible somehow to do it. Be it commands in the CLI, or scripts
using API calls to massage the metadata, or whatever.
--
Peter Wemm - peter@wemm.org; peter@FreeBSD.org; peter@yahoo-inc.com
"All of this is for nothing if we don't go to the stars" - JMS/B5
"If Java had true garbage collection, most programs would delete
themselves upon execution." -- Robert Sewell
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by Steinar Bang <sb...@dod.no>.
>>>>> "Hyrum K. Wright" <hy...@mail.utexas.edu>:
> We've also kicked around the idea for 'svn sever', which would let you
> detach a part of a working copy. So the operation would still be
> supported, it would just take one more step.
+1 on that.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by "Hyrum K. Wright" <hy...@mail.utexas.edu>.
Oren Eini (Murphy & Associates) wrote:
> Um, do I count as someone who actually breaks working copies?
> That is not a highly used feature, I'll admit, and not one that I'll particulary miss.
We've also kicked around the idea for 'svn sever', which would let you
detach a part of a working copy. So the operation would still be
supported, it would just take one more step.
(Oh, and just a reminder that we bottom post on this list. Not a big
deal, but it does help to the flow of the conversation.)
-Hyrum
> Thanks for clarifying on its optionality.
> ________________________________________
> From: sussman@gmail.com [sussman@gmail.com] On Behalf Of Ben Collins-Sussman [sussman@red-bean.com]
> Sent: Wednesday, April 09, 2008 3:38 PM
> To: Oren Eini (Murphy & Associates)
> Cc: Chris Frost; Edward.Harvney@patni.com; dev@subversion.tigris.org
> Subject: Re: performance enhancement by working copy svn server
>
> On Wed, Apr 9, 2008 at 12:23 PM, Oren Eini (Murphy & Associates)
> <v-...@microsoft.com> wrote:
>> From my perspective, I really like SVN's ability to figure what changed on its own, without having to be told.
>
> So do we all. That's why we've always talked about making the 'svn
> edit' model into an *optional* mode of working. Try using subversion
> with a 700MB source tree; running 'svn status' takes minutes to crawl
> the tree (along with commits and updates).
>
>> And having the metadata localized allows to do wierd things like manually merge WC temporarily, or easily take a smaller piece of the WC and move it elsewhere.
>
> That's what we've been telling ourselves for 8 years; it turns out
> that nobody actually ever breaks working copies into small portable
> pieces. And the price we pay for scattered metadata is huge.
RE: performance enhancement by working copy svn server
Posted by "Oren Eini (Murphy & Associates)" <v-...@microsoft.com>.
Um, do I count as someone who actually breaks working copies?
That is not a highly used feature, I'll admit, and not one that I'll particulary miss.
Thanks for clarifying on its optionality.
________________________________________
From: sussman@gmail.com [sussman@gmail.com] On Behalf Of Ben Collins-Sussman [sussman@red-bean.com]
Sent: Wednesday, April 09, 2008 3:38 PM
To: Oren Eini (Murphy & Associates)
Cc: Chris Frost; Edward.Harvney@patni.com; dev@subversion.tigris.org
Subject: Re: performance enhancement by working copy svn server
On Wed, Apr 9, 2008 at 12:23 PM, Oren Eini (Murphy & Associates)
<v-...@microsoft.com> wrote:
> From my perspective, I really like SVN's ability to figure what changed on its own, without having to be told.
So do we all. That's why we've always talked about making the 'svn
edit' model into an *optional* mode of working. Try using subversion
with a 700MB source tree; running 'svn status' takes minutes to crawl
the tree (along with commits and updates).
> And having the metadata localized allows to do wierd things like manually merge WC temporarily, or easily take a smaller piece of the WC and move it elsewhere.
That's what we've been telling ourselves for 8 years; it turns out
that nobody actually ever breaks working copies into small portable
pieces. And the price we pay for scattered metadata is huge.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by Talden <ta...@gmail.com>.
> > Of course, for those stuck with a slow file system (NTFS, UFS, NFS
> > etc), even a serious WC reorganisation may not help enough. This
> > remains to be seen - an 'svn edit' option would feel like admitting
> > defeat.
>
> Yes, I agree it would "feel like admitting defeat". I have no problem with
> it if it is an option.
>
> Just as a data point:
>
> I work with a 6 Gig working copy (14 Gig repository) and can do status and
> commit and update operations in seconds on Win XP SP2 with 2 Gig of RAM and
> NTFS.
I believe it's more about the number of files and folders than size
for repo crawling costs.
The worst we have is ~20,000 files in 3,500 folders (not counting
working-copy meta-data). The actual content-size is only ~650MB and
half of that size is in 10% of the files.
There's a noticeable difference in status checks between Subversion
and tools like Bazaar which uses not only a single meta-data folder
but also a very small number of files in the meta-data area.
Obviously size of working-copy content is a factor when considering
the space cost of the meta-data in Subversion with pristine copies
taking up a lot of space - those pristine copies are expensive. The
distributed solutions don't help as they include a complete revision
history (even if the repository format is often more concise than
Subversion, since clients don't wear the cost of Subversion
repo-size).
(Then again I have 10 working copies that share most of their ancestry
so a Bazaar shared repository with 10 branches/checkouts would reduce
the cost dramatically per working-copy.)
One way to reduce typical status costs would be to 'schedule' changes
for commit (I added this, removed that, or modified these) and only
looked for other changes when a full status (which would include
unknowns and unscheduled modifications and removes).
Of course that's a major UI change for Subversion and I wouldn't
expect it to be popular... Look at the UI Bazaar where a single 'bzr
add' adds all unknowns and you allow the 'status' to discover
modifications and implied removals - I think people will generally
choose a UI that requires the user to do less data-entry over the cost
of a few seconds.
(of course people also challenge Bazaars performance relative to its
peers Mercurial and git so maybe the cost for that UI is too high for
some - to them I say, try CVS).
NB: I'm not advocating Bazaar here, it's merely a comparison... Bazaar
misses, at least (though I'm no Bazaar expert), EOL support, version
properties, mime-type support, partial branches, nesting (externals or
Mercurials forest), mixed revision branches and tags, file/folder-copy
tracking and any reasonable support for case-insensitive
file-systems...
--
Talden
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
RE: performance enhancement by working copy svn server
Posted by David Summers <da...@summersoft.fay.ar.us>.
Ummm, we don't have "svn edit" currently, so it would have to be without
it. :-)
(Or am I missing someting?)
- David
On Thu, 10 Apr 2008, Oren Eini (Murphy & Associates) wrote:
> With or without svn edit?
> ________________________________________
> From: David Summers [david@summersoft.fay.ar.us]
> Sent: Thursday, April 10, 2008 10:36 AM
> To: Mattias Engdegård
> Cc: dev@subversion.tigris.org
> Subject: Re: performance enhancement by working copy svn server
>
>
> I work with a 6 Gig working copy (14 Gig repository) and can do status and
> commit and update operations in seconds on Win XP SP2 with 2 Gig of RAM and
> NTFS.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: dev-help@subversion.tigris.org
>
>
--
David Wayne Summers "Linux: Because reboots are for hardware upgrades!"
david@summersoft.fay.ar.us PGP Key: http://summersoft.fay.ar.us/~david/pgp.txt
PGP Key fingerprint = 0B44 B118 85CC F4EC 7021 1ED4 1516 5B78 E320 2001
RE: performance enhancement by working copy svn server
Posted by "Oren Eini (Murphy & Associates)" <v-...@microsoft.com>.
With or without svn edit?
________________________________________
From: David Summers [david@summersoft.fay.ar.us]
Sent: Thursday, April 10, 2008 10:36 AM
To: Mattias Engdegård
Cc: dev@subversion.tigris.org
Subject: Re: performance enhancement by working copy svn server
I work with a 6 Gig working copy (14 Gig repository) and can do status and
commit and update operations in seconds on Win XP SP2 with 2 Gig of RAM and
NTFS.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
RE: performance enhancement by working copy svn server
Posted by "Harvey, Edward" <Ed...@patni.com>.
> Just as a data point:
>
> I work with a 6 Gig working copy (14 Gig repository) and can
> do status and commit and update operations in seconds on Win
> XP SP2 with 2 Gig of RAM and NTFS.
Another data point:
I have a location with 20 users on a few machines. At any given time, there are some compute crunching jobs running, which means that the memory cached copy of generally idle .svn directories quickly swap out. Users typically update 1-3 times a day each. Their working copies are 2.1G including .svn dirs, and 43,000 files not counting .svn dirs. Each time they svn up, it takes 7-12 minutes. But if they immediately re-run the svn up, it taked 30-40 seconds. Unfortunately a few hours or a day later, svn up will again require 7-12 minutes.
Files are on local raid-5 disk, with caching controller cards, RHEL 4u5.
This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, e-transmission dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error kindly delete this e-mail from your records. If it appears that this mail has been forwarded to you without proper authority, please notify us immediately at netadmin@patni.com and delete this mail.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by David Summers <da...@summersoft.fay.ar.us>.
On Thu, 10 Apr 2008, Mattias Engdegård wrote:
> "Ben Collins-Sussman" <su...@red-bean.com> writes:
>
>> So do we all. That's why we've always talked about making the 'svn
>> edit' model into an *optional* mode of working. Try using subversion
>> with a 700MB source tree; running 'svn status' takes minutes to crawl
>> the tree (along with commits and updates).
>
> No it doesn't, at least not with the WC on a decent local file system
> and there is RAM to cache enough of it - it is usually a matter of
> seconds or less once heated up. (I frequently develop on tmpfs because
> of this, but ext3 works well too.)
>
> But of course I agree, the current WC organisation scatters
> information in lots and lots of files and directories and with very
> expensive locking; there is a lot to gain by saner arrangements.
>
> Of course, for those stuck with a slow file system (NTFS, UFS, NFS
> etc), even a serious WC reorganisation may not help enough. This
> remains to be seen - an 'svn edit' option would feel like admitting
> defeat.
>
Yes, I agree it would "feel like admitting defeat". I have no problem
with it if it is an option.
Just as a data point:
I work with a 6 Gig working copy (14 Gig repository) and can do status and
commit and update operations in seconds on Win XP SP2 with 2 Gig of RAM and
NTFS.
--
David Wayne Summers "Linux: Because reboots are for hardware upgrades!"
david@summersoft.fay.ar.us PGP Key: http://summersoft.fay.ar.us/~david/pgp.txt
PGP Key fingerprint = 0B44 B118 85CC F4EC 7021 1ED4 1516 5B78 E320 2001
Re: performance enhancement by working copy svn server
Posted by Mattias Engdegård <ma...@virtutech.se>.
"Ben Collins-Sussman" <su...@red-bean.com> writes:
>So do we all. That's why we've always talked about making the 'svn
>edit' model into an *optional* mode of working. Try using subversion
>with a 700MB source tree; running 'svn status' takes minutes to crawl
>the tree (along with commits and updates).
No it doesn't, at least not with the WC on a decent local file system
and there is RAM to cache enough of it - it is usually a matter of
seconds or less once heated up. (I frequently develop on tmpfs because
of this, but ext3 works well too.)
But of course I agree, the current WC organisation scatters
information in lots and lots of files and directories and with very
expensive locking; there is a lot to gain by saner arrangements.
Of course, for those stuck with a slow file system (NTFS, UFS, NFS
etc), even a serious WC reorganisation may not help enough. This
remains to be seen - an 'svn edit' option would feel like admitting
defeat.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by Ben Collins-Sussman <su...@red-bean.com>.
On Wed, Apr 9, 2008 at 12:23 PM, Oren Eini (Murphy & Associates)
<v-...@microsoft.com> wrote:
> From my perspective, I really like SVN's ability to figure what changed on its own, without having to be told.
So do we all. That's why we've always talked about making the 'svn
edit' model into an *optional* mode of working. Try using subversion
with a 700MB source tree; running 'svn status' takes minutes to crawl
the tree (along with commits and updates).
> And having the metadata localized allows to do wierd things like manually merge WC temporarily, or easily take a smaller piece of the WC and move it elsewhere.
That's what we've been telling ourselves for 8 years; it turns out
that nobody actually ever breaks working copies into small portable
pieces. And the price we pay for scattered metadata is huge.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
RE: performance enhancement by working copy svn server
Posted by "Oren Eini (Murphy & Associates)" <v-...@microsoft.com>.
RE: performance enhancement by working copy svn server
Posted by "Harvey, Edward" <Ed...@patni.com>.
> libsvn_wc rewrite which
> (1) centralizes metadata in one place and
> (2) allows users the option of 'svn edit' to tell svn what you're
> changing, so that 'svn status/update/commit' no longer need
> to traverse the tree.
With the wc rewrite, are you talking about somehow preventing extraneous processes, such as "vi somefile" from accessing files in a WC "behind the back" of the svn client? Or just telling users, "don't do that" and assuming during the stat/up/commit that they've behaved properly?
The ideal, which may be very difficult to achieve, is the ability to monitor FS access, to automatically "notice" a file changed on the fly. This extra FS overhead is analogous to journaling, where you spread out the fsck overhead during normal operation, instead of letting it all pile up into one big operation.
This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, e-transmission dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error kindly delete this e-mail from your records. If it appears that this mail has been forwarded to you without proper authority, please notify us immediately at netadmin@patni.com and delete this mail.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by Eric Gillespie <ep...@pretzelnet.org>.
"Ben Collins-Sussman" <su...@red-bean.com> writes:
> As Greg Hudson mentioned, a number of us are really quite enthusiastic
> about a libsvn_wc rewrite which (1) centralizes metadata in one place
> and (2) allows users the option of emulating perforce -- 'svn edit' to
Emulating Perforce isn't something anyone sane wants to do. In
this specific example, 'svn edit' will be a pure-local operation,
and will never be held up by some lock or high load or whatever
on the server. I know you know this, but I couldn't help
reacting to this "emulating perforce" ;->.
--
Eric Gillespie <*> epg@pretzelnet.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by Ben Collins-Sussman <su...@red-bean.com>.
As Greg Hudson mentioned, a number of us are really quite enthusiastic
about a libsvn_wc rewrite which (1) centralizes metadata in one place
and (2) allows users the option of emulating perforce -- 'svn edit' to
tell svn what you're changing, so that 'svn status/update/commit' no
longer need to traverse the tree.
We're simply laying low, waiting for svn 1.5 to finish. When 1.5 is
out, I encourage you to participate in this design. I think it's a
better bet than digging into FUSE-like things.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by Chris Frost <ch...@frostnet.net>.
On Mon, Apr 07, 2008 at 01:09:23PM -0700, Jack Repenning wrote:
> FUSE is a fairly portable (Linux, Mac OS X) basis for such a thing:
> http://fuse.sourceforge.net/
>
> Chris Frost has worked on serving Subversion working copies out of a FUSE
> volume, in his "scord" project:
> http://scord.sourceforge.net/
>
> His goal is different from yours: to reduce the disc storage costs of the
> pristine copies. But he's inserted himself into the file system in the right
> spot to do these other things as well.
Fwiw, I see two relevant parts to the speed of the working copy operations
status, diff, etc:
1) read the necessary data from disk
2) traverse the in-memory file system
Part 1 happens to be what I notice most as limiting my svn use. However, it
sounds like only part 2 is a limitation in your situation. (And you could,
of course, aim to improve both parts.)
Routing working copy access through a FUSE file system would definitely be a
way to go. Fwiw, I think the main three limitations to watch out for would be:
- The FUSE API only works on Linux, MacOSX, and BSD
- Serving large data chunks or running many filesystem system calls via
FUSE incurs a significant increase in the number of kernel-user crossings,
data copy operations, and kernel file system cache memory used
- Supporting multiple users with a FUSE server instance might be hard
And easier, but still something to think about, is server/os crash recovery.
I suspect you might make it safe on file systems like ext3 in its default
journaling mode, ordered journaling, and not safe on other file systems,
as the working copy library does itself. Fwiw, I have considered augmenting
the underlying working copy representation that scord itself stores, but
each time decided that the benefit didn't justify the synchronization work.
But these issues might work differently for this scenario.
A David mentioned, file notification mechanisms, like at least Linux and OSX
have, would be another route. This would mostly-to-completely avoid the speed,
multi-user, and safety FUSE limitations.
--
Chris Frost | <http://www.frostnet.net/chris/>
-------------+----------------------------------
PGP: <http://www.frostnet.net/chris/pgpkey.txt>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by Jack Repenning <jr...@collab.net>.
On Apr 5, 2008, at 4:28 PM, Harvey, Edward wrote:
> So the idea is - Suppose there were a special "working copy" svn
> server, which could have exclusive access to some folder on the
> system, containing the working copy, and then export those files to
> localhost, where they could be mounted into a user's home directory
> (or wherever). The user would work in his/her home directory as
> always, but the "working copy svn server" would be able to monitor
> all file access. No change of behavior would be necessary; the
> working copy could contain versioned and unversioned files exactly
> as it always has, but the "working copy svn server" could keep track
> of all the local modifications on the fly, and eliminate the need to
> ever walk the tree.
FUSE is a fairly portable (Linux, Mac OS X) basis for such a thing:
http://fuse.sourceforge.net/
Chris Frost has worked on serving Subversion working copies out of a
FUSE volume, in his "scord" project:
http://scord.sourceforge.net/
His goal is different from yours: to reduce the disc storage costs of
the pristine copies. But he's inserted himself into the file system in
the right spot to do these other things as well.
-==-
Jack Repenning
Chief Technology Officer
CollabNet, Inc.
8000 Marina Boulevard, Suite 600
Brisbane, California 94005
office: +1 650.228.2562
mobile: +1 408.835.8090
raindance: +1 877.326.2337, x844.7461
aim: jackrepenning
skype: jackrepenning
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by David O'Shea <da...@s3group.com>.
This is similar to how Clearcase works also. Something like inotify
would be helpful too I guess, but that would be linux only.
David.
On 06/04/2008 05:30, Greg Hudson wrote:
> As I understand it, Perforce does this, although I believe it requires
> the file server to be hosted by the repository. It is, in fact, quite
> fast as long as the mount is going over a fast link. It's a bit
> unlikely that Subversion will ever go in this direction since it
> requires a lot of system-dependent code and/or documentation.
>
> Another possibile way to avoid scans is to require an "svn edit" (actual
> syntax may vary) to make Subversion notice an edit to a file. This idea
> is most attractive to people who have Subversion integrated with their
> editor or IDE; since not everyone is in that camp, it would probably
> have to be a per-checkout option. Peering at trunk/notes/wc-ng-design,
> it looks like Subversion might conceivably go in this direction if a WC
> rewrite effort succeeds.
>
> On Sat, 2008-04-05 at 19:28 -0400, Harvey, Edward wrote:
>> Basically, just wondering if anyone has ever thought along these lines ...
>>
>> I was thinking about how to speed up operations, such as svn status, etc, which are currently required to walk the tree. With large numbers of files, this can be expensive. The root cause is simple - peoples files can be changed by any process without the knowledge of svn; therefore svn must inspect each file to see if it's changed. To improve this would require some process to have the ability to constantly monitor all file access within the working copy.
>>
>> So the idea is - Suppose there were a special "working copy" svn server, which could have exclusive access to some folder on the system, containing the working copy, and then export those files to localhost, where they could be mounted into a user's home directory (or wherever). The user would work in his/her home directory as always, but the "working copy svn server" would be able to monitor all file access. No change of behavior would be necessary; the working copy could contain versioned and unversioned files exactly as it always has, but the "working copy svn server" could keep track of all the local modifications on the fly, and eliminate the need to ever walk the tree.
>>
>> Granted, root would have to assist somehow, to let the filesystem mount. And I'm sure it's not a trivial code to write.
>>
>> This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, e-transmission dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error kindly delete this e-mail from your records. If it appears that this mail has been forwarded to you without proper authority, please notify us immediately at netadmin@patni.com and delete this mail.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
>> For additional commands, e-mail: dev-help@subversion.tigris.org
>>
>
>
> ----------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: dev-help@subversion.tigris.org
>
--
The information contained in this e-mail and in any attachments is confidential and is designated solely for the attention of the intended recipient(s). If you are not an intended recipient, you must not use, disclose, copy, distribute or retain this e-mail or any part thereof. If you have received this e-mail in error, please notify the sender by return e-mail and delete all copies of this e-mail from your computer system(s).
Please direct any additional queries to: communications@s3group.com.
Thank You.
Silicon and Software Systems Limited. Registered in Ireland no. 378073.
Registered Office: South County Business Park, Leopardstown, Dublin 18
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: performance enhancement by working copy svn server
Posted by Greg Hudson <gh...@MIT.EDU>.
As I understand it, Perforce does this, although I believe it requires
the file server to be hosted by the repository. It is, in fact, quite
fast as long as the mount is going over a fast link. It's a bit
unlikely that Subversion will ever go in this direction since it
requires a lot of system-dependent code and/or documentation.
Another possibile way to avoid scans is to require an "svn edit" (actual
syntax may vary) to make Subversion notice an edit to a file. This idea
is most attractive to people who have Subversion integrated with their
editor or IDE; since not everyone is in that camp, it would probably
have to be a per-checkout option. Peering at trunk/notes/wc-ng-design,
it looks like Subversion might conceivably go in this direction if a WC
rewrite effort succeeds.
On Sat, 2008-04-05 at 19:28 -0400, Harvey, Edward wrote:
> Basically, just wondering if anyone has ever thought along these lines ...
>
> I was thinking about how to speed up operations, such as svn status, etc, which are currently required to walk the tree. With large numbers of files, this can be expensive. The root cause is simple - peoples files can be changed by any process without the knowledge of svn; therefore svn must inspect each file to see if it's changed. To improve this would require some process to have the ability to constantly monitor all file access within the working copy.
>
> So the idea is - Suppose there were a special "working copy" svn server, which could have exclusive access to some folder on the system, containing the working copy, and then export those files to localhost, where they could be mounted into a user's home directory (or wherever). The user would work in his/her home directory as always, but the "working copy svn server" would be able to monitor all file access. No change of behavior would be necessary; the working copy could contain versioned and unversioned files exactly as it always has, but the "working copy svn server" could keep track of all the local modifications on the fly, and eliminate the need to ever walk the tree.
>
> Granted, root would have to assist somehow, to let the filesystem mount. And I'm sure it's not a trivial code to write.
>
> This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, e-transmission dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error kindly delete this e-mail from your records. If it appears that this mail has been forwarded to you without proper authority, please notify us immediately at netadmin@patni.com and delete this mail.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: dev-help@subversion.tigris.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org