You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Greg Hudson <gh...@MIT.EDU> on 2002/09/25 05:59:52 UTC

Eliminating the text-base penalty

Here's a brain dump on getting rid of the text-base penalty, since
some people expressed interest.

Option 1 is to, given the appropriate checkout option, omit the text
base and instead store a checksum of the working copy file (after
newline and keyword substitution, but with no local modifications).
We can use this checksum to detect whether a working copy file has
changed, if the mod time has budged.

Any operation which needs the text base will have to pull it off the
server if the working copy file has changed.

Option 2 is to implement read-only working copies.  The idea here is
that, given the appropriate checkout option, files are checked out
read-only with no text-base.  When you want to edit a file, you have
to do an "svn edit filename" to make it read-write; this creates a
text-base (either by pulling it from the repository or by
de-substituting it from the working copy; the latter option doesn't
require the network but would require stashing some notes about what
substitutions we performed on each file).  So you only pay a text-base
penalty on files you've chosen to edit.

Read-only working copies have a couple of other applications as well.
One is advisory locking; another is "fast crawl," which I described in
http://subversion.tigris.org/servlets/ReadMsg?list=dev&msgId=184609

Any major post-1.0 cycles I might have are currently reserved for a
direct-mapping filesystem back end, so I don't expect to put effort
into either of these plans.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Eliminating the text-base penalty

Posted by "Kirby C. Bohling" <kb...@birddog.com>.
On Sat, 2002-09-28 at 19:28, Justin Erenkrantz wrote:
> On Sat, Sep 28, 2002 at 05:11:53PM -0700, Jon Watte wrote:
> > I'm sure you understand and agree that Subversion will NOT be useful to a 
> > large organization/project until it implements this mode of operation. On 
> > the flip side, I entirely understand that the CVS model is initially much 
> > more convenient for programmers in small to medium sized projects, and 
> > the transition is just one of many you have to go through in order to be 
> > able to work on a larger scale.
> 

	Jon's worried about scaling in terms of files in the repository.  That
at least is his problem if I understand it correctly.  He will have his
problems if he is on the only user of the CVS.  It'll still take 10
minutes to do the stat().  I have a hard time believing even the crappy
filesystems under windows take that long, but I can believe 10 minutes
waiting on CVS.  Maybe over a network fs, but that's still a really long
time.  I can stat all 250K files on my Linux box in under 2 minutes w/ a
crappy IDE drive.

	He'd like it if he has on line changes 5 files in 4 different
directories, he'd like to not have to pay a penalty because there are
20K files in it.  It's only 5 edits.  It'd be great if commit time was
proportional to the number of files edited, not proportional to the
number of files in the repository.

> I think you have your models reversed.  A reserved checkout model has
> horrible scaling problems - people end up tripping over themselves
> because they are doing parallel development.  User K in Chicago
> 'reserved' a file and didn't finish his changes or went on vacation,
> but User G in SF wants to do another change in that same file while
> User K holds the lock.  Oops.  So, you force User G to break the lock.
> 

	You are discussing the number of people in the who have access to the
project.  Yes if 100 people are all working on a single file, resevered
check-outs is very bad.  It's a problem even if there are only a handful
of files.  But that's not the problem Jon's discussing.


> Reserved checkouts may make more sense in smaller projects (where
> ownership is well-defined), but almost certainly has no place in
> large projects where ownership isn't as well-defined or
> enforced.  -- justin
> 

	Presumably, you'd like a solution that scales in both directions.  That
or a solution that will let the user choose which direction they need to
scale in more.  For Jon, clearly he needs something that will scale in
terms of file count.  Where as large development groups will need
something that will scale in terms of number of concurrent writers to a
file.  

	If a user is willing to take reponsibility for doing "svn edit" before
they change a file why do you tell them it won't work.  If that were
true, Perforce wouldn't be a viable product.  Clearly people can and do
get use to it.  I'd hate it, but I'd get over it, if it saved me 10
minutes everytime I did an svn diff/commit.


	Thanks,
		Kirby

> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: dev-help@subversion.tigris.org
> 
-- 
Real Programmers view electronic multimedia files with a hex editor.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

RE: Eliminating the text-base penalty

Posted by Sander Striker <st...@apache.org>.
> From: Justin Erenkrantz [mailto:jerenkrantz@apache.org]
> Sent: 29 September 2002 03:43

[...]
> Because we're going to have to offer both models - the CVS model and
> the Perforce model.  You're going to miss the point of SVN if we
> decide to abandon the CVS model and enforce everyone to do 'svn edit'
> before letting them modify the file.  So, it must be an alternative
> and one that can't mix with checkouts using the CVS model.  -- justin

A simple option in the svn config file would take care of that:

  mode = [require_edit|normal]  (poorly chosen values, but hey)

This mode of operation, client side, is something that is user specific
anyway  (maybe working copy specific, but you get the idea).

Sander



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Eliminating the text-base penalty

Posted by Justin Erenkrantz <je...@apache.org>.
On Sat, Sep 28, 2002 at 06:06:59PM -0700, Kevin Pilch-Bisson wrote:
> Perforce is not the filesystem.  Maybe you are thinking of clearcase.  Here's 
> how perforce works.

Indeed, I was confused.  =)  Sorry, they all run together.

> If you forget (hard to do with the read-only thing), there is a command which 
> will look at your working copy and check for any files that have been changed 
> to be writable, and decide that you wanted to edit them.  

Which is what you have to introduce to work around the fact that
the version control system can't control the working copy.  When
it is exposed as a file system, it's much easier to handle.
(Indeed, I'd imagine complete DAV-based file systems would operate
like this - flipping write attr on gets a LOCK.)

And, since this would be optional in SVN (see below), the failure
mode is a lot easier to hit.  Perhaps, you could add an
--advisory-edit option to checkout, which would enforce that the WC
is checked-out read-only and you can use 'svn edit' to flip it to
write access just for that file.  But, if you sustain the current
model of write-access checkouts and then using 'svn edit,' you're
going to screw up more often that not.

The problem is that if we introduce advisory edit functionality
(or an interface that does), that same interface is what is going to
*have* to implement advisory server-level locking.  If you want to
suggest that we have to do:

svn local-edit foo        or  svn edit foo
svn server-edit foo           svn lock foo

Ugh.  So, while you might think the server isn't involved now, it's
going to *have* to be when we introduce advisory locking.  I think
the options (from the user perspective) mean the same thing - that
I intend to modify this file - hence, we should do whatever we
have to for this file.  In most cases, the user doesn't care where
that happens - client-side or server-side or both.

So, that can mean that we locally note that this file is going to be
modified, but on a RA that supports locking, that also means that
the lock has to be taken out.  As a side-effect of holding the
advisory lock, the client can be smart and skip the stat() calls
for those files that don't have a lock.

Yes, you could attempt to add a "--no-server" option to edit
which won't grab the advisory lock on the server, but I'd argue
that's not what the user wants - if the WC is by default read-only,
then the repository should require locks to modify.  'svn lock foo'
could also run 'svn edit foo' if needed, but that's ugly, too.  So,
I think these items are incredibly intertwined.

> Yes there are failure modes, but I don't see why they are troublesome.  

Because we're going to have to offer both models - the CVS model and
the Perforce model.  You're going to miss the point of SVN if we
decide to abandon the CVS model and enforce everyone to do 'svn edit'
before letting them modify the file.  So, it must be an alternative
and one that can't mix with checkouts using the CVS model.  -- justin

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Eliminating the text-base penalty

Posted by Kevin Pilch-Bisson <ke...@pilch-bisson.net>.
Quoting Justin Erenkrantz <je...@apache.org>:

> On Sat, Sep 28, 2002 at 08:31:58PM -0400, Garrett Rooney wrote:
> > but he's not asking for reserved checkouts, he's just asking for some 
> > model where the client already knows what files you are editing, so it 
> > doesn't have to stat everything to check when you want to diff or 
> > commit or whatever.
> 
> And, that can't work, either.  You have to prevent anyone from
> circumventing the process.  And, the only way to do is to take
> perforce's model of becoming the file system or enforce components
> of the development environment.  (Some work is being done here on
> adding development tools for precisely this level of awareness.)

Perforce is not the filesystem.  Maybe you are thinking of clearcase.  Here's 
how perforce works.

You checkout a working copy.  This works similarly to svn, except that all the 
files are marked read-only, and there is no metadata dir.  You then p4 edit 
each file you want to edit, and the server records that, and makes the file 
writable.

If you forget (hard to do with the read-only thing), there is a command which 
will look at your working copy and check for any files that have been changed 
to be writable, and decide that you wanted to edit them.  

The suggestion is to due the same thing with svn, except keep the metadata 
client side instead of server side. 

Thus the svn edit command, would copy a file to its text-base location, and 
make the source writable.  There would have to be a way to recover from times 
when you forgot to "svn edit" the file, which would probably mean going to the 
server to grab the text-base.
> 
> > this information /might/ be kept server side (like in perforce), but 
> > that doesn't mean that the fact that i'm editing a file means nobody 
> > else can.  knowledge of what i'm working on != locks.
> 
> Any model based on pro-active usage of 'svn edit' is going to
> be prone to failure conditions where modified files will not be
> detected.  Therefore, the only way to correctly identify if a file
> is changed is to do a diff/crc/etc of the on-disk file with what we
> already know about the file.  Since SVN isn't the file system, we
> must admit that we will have no knowledge of when a file changed.
> 
> This failure mode is more troublesome than any speed improvements.
> Entry caching, of course, makes SVN way faster than it has been
> before.  So, I don't buy that large project WCs can't scale now.
> 
> If you want to tout this based on doing reserved checkouts, fine,
> but as far as 'speed improvements' - it's a bad idea.  And, as
> I stated, I don't believe reserved checkouts scale.  -- justin

Yes there are failure modes, but I don't see why they are troublesome.  

"svn scan" could check for any files in the working copy which don't match the 
stored checksum, and see whether there was a corresponding text-base file or 
not.  If not then it grabs one from the server.

Having worked with perforce, the failure mode is possible, but easily 
understood, and not that difficult to recover from.
--
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Kevin Pilch-Bisson
kevin@pilch-bisson.net
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Eliminating the text-base penalty

Posted by Garrett Rooney <ro...@electricjellyfish.net>.
On Saturday, September 28, 2002, at 08:47 PM, Justin Erenkrantz wrote:

> On Sat, Sep 28, 2002 at 08:31:58PM -0400, Garrett Rooney wrote:
>> but he's not asking for reserved checkouts, he's just asking for some
>> model where the client already knows what files you are editing, so it
>> doesn't have to stat everything to check when you want to diff or
>> commit or whatever.
>
> And, that can't work, either.  You have to prevent anyone from
> circumventing the process.  And, the only way to do is to take
> perforce's model of becoming the file system or enforce components
> of the development environment.  (Some work is being done here on
> adding development tools for precisely this level of awareness.)

(note, i don't particularly care one way or another about this feature, 
i hate having to do 'p4 edit' when i use perforce at work, and it seems 
like larger projects should be able to structure their repository and 
build structure such that the developers can check out smaller parts 
rather than the whole thing.)

that said, i don't think we have to make it 'perfect'.  if the client 
actively works around the version control system, then hell with them.  
what's the worst that can happen?  they start editing a file without 
telling svn (if we're making all files read only, they chmod it or 
something), then, later, they do an 'svn commit', the file doesn't get 
committed because svn doesn't think it's change.  their changes break 
the build or cause bugs or whatever, and they get yelled at by the 
other people in the project.  good, they got what they deserved.

>> this information /might/ be kept server side (like in perforce), but
>> that doesn't mean that the fact that i'm editing a file means nobody
>> else can.  knowledge of what i'm working on != locks.
>
> Any model based on pro-active usage of 'svn edit' is going to
> be prone to failure conditions where modified files will not be
> detected.  Therefore, the only way to correctly identify if a file
> is changed is to do a diff/crc/etc of the on-disk file with what we
> already know about the file.  Since SVN isn't the file system, we
> must admit that we will have no knowledge of when a file changed.
>
> This failure mode is more troublesome than any speed improvements.
> Entry caching, of course, makes SVN way faster than it has been
> before.  So, I don't buy that large project WCs can't scale now.

i agree that we should be fast enough to compete without this, but if 
people absolutely want this feature, i don't think 'the user can work 
around it' is a good enough reason to say no.  the user can already 
mess with the version control tool in about 17 different ways by 
editing the files in the .svn dir, we just try to make that as hard as 
possible by making them read-only, etc.  i don't see how this is 
different.

-garrett

-- 
garrett rooney                    Remember, any design flaw you're
rooneg@electricjellyfish.net      sufficiently snide about becomes
http://electricjellyfish.net/     a feature.       -- Dan Sugalski


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

RE: Eliminating the text-base penalty

Posted by Jon Watte <hp...@mindcontrol.org>.
> Hmm. On my box, creating a tree with about 36k nodes takes
> approximately 8.5 seconds; crawling it and statting every node
> takes approximately 4 seconds. If it really takes 10 minutes to
> traverse a tree containing 20k files looking to see what's changed,
> then I suspect the problem is with the implementation, not with
> the principle.

That's the numbers I've seen on Windows with CVS. Of course, CVS does 
more than just stat each file. However, SVN has a similar model.

> Further, most checkins don't need to check the entire tree for
> changed files, nor even a substantial fraction of the entire tree.
> Usually you're working in a directory somewhere near the leaves,
> with (if you're unlucky) 1000 files under it. No?

But my change involves some database schema, which lives in one place; 
some sources and headers, which live elsewhere; some xslt, some images, 
some perl scripts and a new regression test. So, no, my change isn't 
contained in a single directory. A problem that happened a lot with 
CVS was that users would try to submit in "the main directory" and 
forget some other files they've edited. "Educating the users" isn't 
sufficient because these are clever, well-meaning programmers already, 
whose only fault is that they're human.

> is being done wrong in Subversion. And if you often have to
> do a checkin for which Subversion needs to look at 20k files,
> then I think something's wrong in the organization of your
> project. I am open to correction on both issues.

I've found that any project that's big enough will run into these kinds 
of issues. It's just a fact of scale.

> My experiment was admittedly a very simple-minded one,

Did you run it on Windows? Were the files large enough to look like 
real source files?

> This was on a local filesystem on a FreeBSD box.

FreeBSD: Hardly the worst case scenario :-)

Cheers,

				/ h+


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Eliminating the text-base penalty

Posted by Greg Hudson <gh...@MIT.EDU>.
On Sun, 2002-09-29 at 11:20, Gareth McCaughan wrote:
> Why use a working copy that's AFS-mounted over your
> broadband link? Why not have a local working copy at
> home, and talk to a Subversion server at work?

Because I don't want to have one working directory when I'm working from
home and a different one when I'm working from work.  (And my uplink
bandwidth from home is miserable, of course, so accessing a home working
directory from work would be terrible.)  Subversion doesn't change
anything about that.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Eliminating the text-base penalty

Posted by Gareth McCaughan <Ga...@pobox.com>.
On Sunday 29 September 2002 3:35 pm, Greg Hudson wrote:

> I think you guys are considering only the easy cases.

Or, to put it differently,

  | My experiment was admittedly a very simple-minded one,
  | and represents a best case in a few ways.

Haven't I read that somewhere before? :-)

>                                                        My job's
> source tree has 100K or so nodes, and it takes well over half an
> hour to stat them over AFS over a cable modem.

"Doctor, it hurts when I do this."

Why use a working copy that's AFS-mounted over your
broadband link? Why not have a local working copy at
home, and talk to a Subversion server at work?
Of course, you're using CVS right now, and maybe
mounting your working copy is the right way to go
with CVS; but surely it isn't with Subversion.

> That said, we use CVS, and we're not about to run out and use
> Perforce for fast crawl capability.  Getting a list of modified
> files in my whole working dir quickly would be nice sometimes, but
> usually I'm working in a small subset of the tree.

Or, to put it differently,

  | Further, most checkins don't need to check the entire tree for
  | changed files, nor even a substantial fraction of the entire tree.
  | Usually you're working in a directory somewhere near the leaves,
  | with (if you're unlucky) 1000 files under it. No?

I think I read *that* somewhere before, too. :-)

-- 
g

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

RE: Eliminating the text-base penalty

Posted by Greg Hudson <gh...@MIT.EDU>.
On Sun, 2002-09-29 at 07:31, Barry Scott wrote:
> local 100ATA disk: 3.75 seconds 
> samba share from 600MHz PIII running FreeBSD: 50.25 seconds
> 
> stat does not seem to be the problem even on network drive.

I think you guys are considering only the easy cases.  My job's source
tree has 100K or so nodes, and it takes well over half an hour to stat
them over AFS over a cable modem.

That said, we use CVS, and we're not about to run out and use Perforce
for fast crawl capability.  Getting a list of modified files in my whole
working dir quickly would be nice sometimes, but usually I'm working in
a small subset of the tree.

If we already have read-only checkouts because we want to provide locks,
then fast crawl is probably reasonable to support (not too much
complexity for the benefit).  But it's definitely not a 1.0 feature.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

RE: Eliminating the text-base penalty

Posted by Barry Scott <ba...@ntlworld.com>.
>     build("foo", 10, 3, 8)
> 

I ran your test on A 800MHz PIII Windows/XP testing the stat with

dir foo /s >a

local 100ATA disk: 3.75 seconds 
samba share from 600MHz PIII running FreeBSD: 50.25 seconds

stat does not seem to be the problem even on network drive.

> is being done wrong in Subversion. And if you often have to
> do a checkin for which Subversion needs to look at 20k files,
> then I think something's wrong in the organization of your
> project. I am open to correction on both issues.

I agree that its  poor organisation to require that number of
files. But I'll observe that many projects are poorly organised
and SVN has to decide how it will help these projects, best
practise docs and/or optimisations for these large file set
use-cases.

BArry



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Eliminating the text-base penalty

Posted by Gareth McCaughan <Ga...@pobox.com>.
Jon Watte wrote:

> + The problem we're wanting to solve is that it takes 10 minutes to
>   crawl through 20,000 files, trying to figure out which ones have
>   changed when you want to commit (or similar).

Hmm. On my box, creating a tree with about 36k nodes takes
approximately 8.5 seconds; crawling it and statting every node
takes approximately 4 seconds. If it really takes 10 minutes to
traverse a tree containing 20k files looking to see what's changed,
then I suspect the problem is with the implementation, not with
the principle.

Further, most checkins don't need to check the entire tree for
changed files, nor even a substantial fraction of the entire tree.
Usually you're working in a directory somewhere near the leaves,
with (if you're unlucky) 1000 files under it. No?

I don't have any objection to the proposal, but I'm puzzled
by the apparent need for it. If it takes 10 minutes to find
which files, out of 20k, have changed, then I think something
is being done wrong in Subversion. And if you often have to
do a checkin for which Subversion needs to look at 20k files,
then I think something's wrong in the organization of your
project. I am open to correction on both issues.

                                *

My experiment was admittedly a very simple-minded one,
and represents a best case in a few ways. I created a
tree of directories and empty files as follows:

    def build(dir, n_files, n_dirs, depth):
      os.mkdir(dir)
      for i in range(n_files):
        open(os.path.join(dir, str(i)), "w").close()
      depth -= 1
      if depth <= 0: return
      for j in range(n_files, n_files+n_dirs):
        build(os.path.join(dir, str(j)), n_files, n_dirs, depth)

    build("foo", 10, 3, 8)

and I crawled it doing a stat on each node as follows:

    time find foo -ls > /dev/null

This was on a local filesystem on a FreeBSD box.
Hardware: Athlon/1GHz, 256Mb. FreeBSD's filesystem
is quite fast, and the hardware -- though nowhere near
today's bleeding edge -- is quite decent. But, still,
NFS on a 300MHz Ultrasparc (say) surely can't be
more than (say) 25 times slower, which would be
less than 2 minutes.

If statting 2*20k files takes 10 minutes then you're statting
about 70 files a second. That's very, very, very slow.

-- 
g

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

RE: Eliminating the text-base penalty

Posted by Jon Watte <sv...@mindcontrol.org>.
> > but he's not asking for reserved checkouts, he's just asking for some 
> > model where the client already knows what files you are editing, so it 
> > doesn't have to stat everything to check when you want to diff or 
> > commit or whatever.
> 
> And, that can't work, either.  You have to prevent anyone from
> circumventing the process.  And, the only way to do is to take

I don't think you understand the suggestion. All state that's kept is 
local. If you want to not use this model, you're welcome not to -- the 
draw-back for you will be that subversion has to crawl through your 
entire source tree to figure out what needs to be done when you go to 
check in -- i e, the current behavior.

Note that the suggestion on the table is that the user, on initial 
check-out (or subsequent special command) decides which of these modes 
he wants to work in -- again, this is all about using client-side state 
to accelerate time-consuming operations.

> perforce's model of becoming the file system or enforce components
> of the development environment.  (Some work is being done here on
> adding development tools for precisely this level of awareness.)

You must be confusing Perforce with ClearCase. Perforce does not enforce 
any components (other than itself, naturally). However, perforce keeps 
server-side state, which is good for what they do, but means that you 
can't mv or cp or rm your client-side tree without telling the server. 
The suggestion I'm making does not suffer from these problems.

> Any model based on pro-active usage of 'svn edit' is going to
> be prone to failure conditions where modified files will not be
> detected.

All really large projects in the world use the "open to edit" model, 
AFAIK. The CVS model just doesn't scale large enough. If you think 
you can't help yourself but to override that warning your editor will 
give you when trying to overwrite a read-only file, then you are 
perfectly welcome to not turn on the client-side edit list optimization, 
and instead wait the extra 10 minutes on "status", "diff" or "checkin".

> already know about the file.  Since SVN isn't the file system, we
> must admit that we will have no knowledge of when a file changed.

If that's the only model you want to work with, then you are forever 
relegated to small (or possibly medium) size projects. Note 
that the Linux kernel is, at the MOST, a smallish-medium size project.

> before.  So, I don't buy that large project WCs can't scale now.

Been there, done that, made the measurements. If you don't believe me, 
then: whatever.

> If you want to tout this based on doing reserved checkouts, fine,

Reserved check-outs are sometimes necessary for workflow reasons (such 
as preventing artists editing the same 3dsmax file at the same time) so 
it should be _supported_. But that's another feature suggestion that 
I'm not prepared to discuss at this time.

> but as far as 'speed improvements' - it's a bad idea.  And, as
> I stated, I don't believe reserved checkouts scale.  -- justin

It seems to me that you have not yet comprehended the actual suggestion 
under consideration. Just as I (and others) said in previous messages: 
nobody is talking about reserved check-outs.

Let me re-summarize the suggestion:

+ The problem we're wanting to solve is that it takes 10 minutes to 
  crawl through 20,000 files, trying to figure out which ones have 
  changed when you want to commit (or similar).
+ The proposed solution involves keeping local state about which files 
  have actually been modified, and ONLY look at those files.
+ Because SVN is not the file system, it can't know which files have 
  been locally modified, so you have to tell it.
+ Therefore, the user can choose on check-out to put all files in read-
  only mode, and use "svn edit" to set the writable bit and add the file 
  to the local list of files-to-examine-when-checking-in.
+ Once a file has been checked in or reverted, the file is removed from 
  the local edited-files list.
+ Ideally, there is ONE local edited-files list for an entire checked-
  out repository, kept conceptually at the root. That way, you don't 
  even have to crawl the directory tree to find multiple lists of 
  edited files.

Your argument that if the user says "I want to edit this" he will 
automatically also want to lock the file does not hold -- the default 
for Perforce is to not lock, and most people are happy with that. You 
only lock (totally separate feature) if you really need to.


					/ h+

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

RE: Eliminating the text-base penalty

Posted by Jon Watte <hp...@mindcontrol.org>.
> to implement. I still think an entries file would be worth
> keeping around for other potential optimizations though. But
> the notion of only storing the duplicate file when you do
> "svn edit" is a good one.

Thanks. I'm okay with a general "entries" file although that will 
scale slightly worse. Ideally, there will be ONE "entries" file 
for an entire hierarchy of checked-out folders, stored at the 
root. Each directory could point at where it's check-out root 
(or "entries" file) actually lives.

Note that I'm not advocating dropping the current mode of crawling 
the entire repository to generate check-ins. I think both modes 
need to be supported, to cater both to small and large projects.

> As for your comment about Moore's law, I agree in principle.
> But I don't think we should go out of our way to prove it :)

Fair enough. But if it saves time to take it on faith, then it's 
often a low-risk assumption (again, with "linear growth" being the 
keyword here).

Cheers,

				/ h+


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

RE: Eliminating the text-base penalty

Posted by Kean Johnston <jk...@caldera.com>.
> Anyway, if you don't want to store the working copy for all 
> files on the disk, you could make a writable working copy
> (and/or save away the original file) when the user does 
> "svn edit". Then, you could still get diffs and reverts for
> free, with no repository access, and the overhead would only
> be proportional to the amount of files being edited.
I like that notion too. In fact that may make it even easier
to implement. I still think an entries file would be worth
keeping around for other potential optimizations though. But
the notion of only storing the duplicate file when you do
"svn edit" is a good one.

As for your comment about Moore's law, I agree in principle.
But I don't think we should go out of our way to prove it :)

Kean


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

RE: Eliminating the text-base penalty

Posted by Jon Watte <sv...@mindcontrol.org>.
I'm actually not against storing a local copy. The way I see it, that's
just a constant overhead factor (linear in size of repository). Anything
that just grows linearly, and in space (not time), is going to be drowned
out by Moore's law sooner or later anyway.

Anyway, if you don't want to store the working copy for all files on the
disk, you could make a writable working copy (and/or save away the original
file) when the user does "svn edit". Then, you could still get diffs and
reverts for free, with no repository access, and the overhead would only
be proportional to the amount of files being edited.

And if you run in the old-skool CVS mode where everything's always open
for edit, you pay the 2x overhead and longer check-in times, which is just
the current status quo.

Cheers,

			/ h+

> -----Original Message-----
> From: Kean Johnston [mailto:jkj@caldera.com]
> Sent: Saturday, September 28, 2002 9:38 PM
> To: 'Justin Erenkrantz'; 'Garrett Rooney'
> Cc: 'Jon Watte'; 'Greg Stein'; dev@subversion.tigris.org
> Subject: RE: Eliminating the text-base penalty
>
>
> > Any model based on pro-active usage of 'svn edit' is going to
> > be prone to failure conditions where modified files will not be
> > detected.  Therefore, the only way to correctly identify if a file
> > is changed is to do a diff/crc/etc of the on-disk file with what we
> > already know about the file.  Since SVN isn't the file system, we
> > must admit that we will have no knowledge of when a file changed.
> I agree. But you don't need a local copy of the file to achieve this.
> If on checkout, you record the file properties (date/time/mode/crc)
> in an entries list, that's all you need. With this, you have the
> files all locally, and then an "svn edit" can be a simple matter
> of marking the entries file with a flag that says its been edited.
> At checkin time, all you need to do is scan through each entries
> file in each directory, check for any such flags, and delta in those
> files that have changed. It is useful to store a CRC so that if you
> do an "svn edit", but then change your mind after NOT changing the
> file, you can still "svn revert" to remove the modified flag. When
> you revert, you can compare the CRCs to make sure that the file
> really hasn't changed.
>
> Don't forget my original objection to the notion of cached files
> was how badly it scales. Yes, disk is cheap but not that cheap that
> we should encourage wasting it when perfectly good mechanisms for
> achieving most of the same functionality can be done without
> duplicating every file. About the only thing you CANT do with a
> simple fileprops/CRC mechaism is roll back an already edited file
> without contacting the repository. When your repository is 8 or
> more gigabytes, that's a lot of space "wasted" if everything is
> duplicated.
>
> Kean
>


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

RE: Eliminating the text-base penalty

Posted by Kean Johnston <jk...@caldera.com>.
> Any model based on pro-active usage of 'svn edit' is going to
> be prone to failure conditions where modified files will not be
> detected.  Therefore, the only way to correctly identify if a file
> is changed is to do a diff/crc/etc of the on-disk file with what we
> already know about the file.  Since SVN isn't the file system, we
> must admit that we will have no knowledge of when a file changed.
I agree. But you don't need a local copy of the file to achieve this.
If on checkout, you record the file properties (date/time/mode/crc)
in an entries list, that's all you need. With this, you have the
files all locally, and then an "svn edit" can be a simple matter
of marking the entries file with a flag that says its been edited.
At checkin time, all you need to do is scan through each entries
file in each directory, check for any such flags, and delta in those
files that have changed. It is useful to store a CRC so that if you
do an "svn edit", but then change your mind after NOT changing the
file, you can still "svn revert" to remove the modified flag. When
you revert, you can compare the CRCs to make sure that the file
really hasn't changed.

Don't forget my original objection to the notion of cached files
was how badly it scales. Yes, disk is cheap but not that cheap that
we should encourage wasting it when perfectly good mechanisms for
achieving most of the same functionality can be done without
duplicating every file. About the only thing you CANT do with a
simple fileprops/CRC mechaism is roll back an already edited file
without contacting the repository. When your repository is 8 or
more gigabytes, that's a lot of space "wasted" if everything is
duplicated.

Kean


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Eliminating the text-base penalty

Posted by Justin Erenkrantz <je...@apache.org>.
On Sat, Sep 28, 2002 at 08:31:58PM -0400, Garrett Rooney wrote:
> but he's not asking for reserved checkouts, he's just asking for some 
> model where the client already knows what files you are editing, so it 
> doesn't have to stat everything to check when you want to diff or 
> commit or whatever.

And, that can't work, either.  You have to prevent anyone from
circumventing the process.  And, the only way to do is to take
perforce's model of becoming the file system or enforce components
of the development environment.  (Some work is being done here on
adding development tools for precisely this level of awareness.)

> this information /might/ be kept server side (like in perforce), but 
> that doesn't mean that the fact that i'm editing a file means nobody 
> else can.  knowledge of what i'm working on != locks.

Any model based on pro-active usage of 'svn edit' is going to
be prone to failure conditions where modified files will not be
detected.  Therefore, the only way to correctly identify if a file
is changed is to do a diff/crc/etc of the on-disk file with what we
already know about the file.  Since SVN isn't the file system, we
must admit that we will have no knowledge of when a file changed.

This failure mode is more troublesome than any speed improvements.
Entry caching, of course, makes SVN way faster than it has been
before.  So, I don't buy that large project WCs can't scale now.

If you want to tout this based on doing reserved checkouts, fine,
but as far as 'speed improvements' - it's a bad idea.  And, as
I stated, I don't believe reserved checkouts scale.  -- justin

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Eliminating the text-base penalty

Posted by Garrett Rooney <ro...@electricjellyfish.net>.
On Saturday, September 28, 2002, at 08:28 PM, Justin Erenkrantz wrote:

> On Sat, Sep 28, 2002 at 05:11:53PM -0700, Jon Watte wrote:
>> I'm sure you understand and agree that Subversion will NOT be useful 
>> to a
>> large organization/project until it implements this mode of 
>> operation. On
>> the flip side, I entirely understand that the CVS model is initially 
>> much
>> more convenient for programmers in small to medium sized projects, and
>> the transition is just one of many you have to go through in order to 
>> be
>> able to work on a larger scale.
>
> I think you have your models reversed.  A reserved checkout model has
> horrible scaling problems - people end up tripping over themselves
> because they are doing parallel development.  User K in Chicago
> 'reserved' a file and didn't finish his changes or went on vacation,
> but User G in SF wants to do another change in that same file while
> User K holds the lock.  Oops.  So, you force User G to break the lock.

but he's not asking for reserved checkouts, he's just asking for some 
model where the client already knows what files you are editing, so it 
doesn't have to stat everything to check when you want to diff or 
commit or whatever.

this information /might/ be kept server side (like in perforce), but 
that doesn't mean that the fact that i'm editing a file means nobody 
else can.  knowledge of what i'm working on != locks.

-garrett

-- 
garrett rooney                    Remember, any design flaw you're
rooneg@electricjellyfish.net      sufficiently snide about becomes
http://electricjellyfish.net/     a feature.       -- Dan Sugalski


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

RE: Eliminating the text-base penalty

Posted by Jon Watte <hp...@mindcontrol.org>.
> > more convenient for programmers in small to medium sized projects, and 
> > the transition is just one of many you have to go through in 
> order to be 
> > able to work on a larger scale.

> I think you have your models reversed.  A reserved checkout model has
> horrible scaling problems - people end up tripping over themselves
> because they are doing parallel development.  User K in Chicago

I said nothing about "reserved". I said "telling the system what you're 
editing". When the system has to stat 20,000 files to figure out what 
changed, that can take 10 minutes on typical Windows file systems. If 
the system already has a list of the files that you have changed, because 
you told the system, then it's instantaneous.

> Reserved checkouts may make more sense in smaller projects (where

I am not advocating reserved check-outs. Indeed, I'm advocating something 
which could (should!) be implemented solely using client-side state, to 
greatly accelerate operations that need to know which files have been 
changed.

Cheers,

				/ h+


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Eliminating the text-base penalty

Posted by Justin Erenkrantz <je...@apache.org>.
On Sat, Sep 28, 2002 at 05:11:53PM -0700, Jon Watte wrote:
> I'm sure you understand and agree that Subversion will NOT be useful to a 
> large organization/project until it implements this mode of operation. On 
> the flip side, I entirely understand that the CVS model is initially much 
> more convenient for programmers in small to medium sized projects, and 
> the transition is just one of many you have to go through in order to be 
> able to work on a larger scale.

I think you have your models reversed.  A reserved checkout model has
horrible scaling problems - people end up tripping over themselves
because they are doing parallel development.  User K in Chicago
'reserved' a file and didn't finish his changes or went on vacation,
but User G in SF wants to do another change in that same file while
User K holds the lock.  Oops.  So, you force User G to break the lock.

Reserved checkouts may make more sense in smaller projects (where
ownership is well-defined), but almost certainly has no place in
large projects where ownership isn't as well-defined or
enforced.  -- justin

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

RE: Eliminating the text-base penalty

Posted by Jon Watte <hp...@mindcontrol.org>.
> Agreed, but it doesn't follow the "normal model" for CVS/SVN operation. I
> could easily see it as a future extension, so much so, that I'd 
> say that the
> model *will* be available at some point. And when it does, then these

That's encouraging.

> Until then, requiring users to "svn edit" *before* work is a bit of a
> stretch from the current model. While we could superficially do 
> such a thing
> quite quickly, I'd worry that we'll start missing holes here and there.

I'm sure you understand and agree that Subversion will NOT be useful to a 
large organization/project until it implements this mode of operation. On 
the flip side, I entirely understand that the CVS model is initially much 
more convenient for programmers in small to medium sized projects, and 
the transition is just one of many you have to go through in order to be 
able to work on a larger scale.

Cheers,

				/ h+


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Eliminating the text-base penalty

Posted by Greg Stein <gs...@lyra.org>.
On Fri, Sep 27, 2002 at 11:29:42AM -0700, Jon Watte wrote:
>...
> To put it another way: the keeping of state telling the system which files
> are actually open (and the general status of each file) in a place where
> getting at it is "instantaneous" accelerates working on large code bases
> by order(s) of magnitude. Perforce does this server-side. I think that most
> of the benefits could be had by doing it client-side, too, which would fit
> better in the SVN model.

Agreed, but it doesn't follow the "normal model" for CVS/SVN operation. I
could easily see it as a future extension, so much so, that I'd say that the
model *will* be available at some point. And when it does, then these
questions can come back up, and the code can be beat into submission for
some great performance gains.

Until then, requiring users to "svn edit" *before* work is a bit of a
stretch from the current model. While we could superficially do such a thing
quite quickly, I'd worry that we'll start missing holes here and there.

Cheers,
-g

-- 
Greg Stein, http://www.lyra.org/

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Eliminating the text-base penalty

Posted by Ask Bjoern Hansen <as...@develooper.com>.
On Fri, 27 Sep 2002, Rafael Garcia-Suarez wrote:

> However -- I use Perforce on a project where some generated files
> are also kept in the repository. That's means that when those files
> are to be regenerated, we've to p4-edit them before running the
> generation program.

[perforce stuff; sorry]

It's a bit of a hack, but you don't have to open everything for edit
for that kind of thing.

chmod -R +w apps
# do the work
p4 diff -sd | p4 -x - delete
p4 diff -se | p4 -x - edit
(find ./apps/ -type f | p4 -x - files 2>&1 >/dev/null)  | cut '-d ' -f1 > /tmp/newfiles
vi /tmp/newfiles
p4 -x - add < /tmp/newfiles
p4 submit
chmod -R -w apps

Or if that's too ugly then I never found it too much hassle to do a
p4 edit of everything and then use "p4 diff -sr" to find files to
revert before doing p4 submit.


 - ask (who doesn't use perforce anymore)

-- 
ask bjoern hansen, http://www.askbjoernhansen.com/ !try; do();


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

RE: Eliminating the text-base penalty

Posted by Jon Watte <hp...@mindcontrol.org>.
> > to inspect for "diff" or "checkin" operations -- only actually edited
> > files are actually considered; no stat-ing of 18,000 files just to
> > figure out what to do.
>
> Yes, for performance reasons that could be useful.

It's actually more in the "life saver" category than just "useful" for us.

> However -- I use Perforce on a project where some generated files
> are also kept in the repository. That's means that when those files

Well, that's your problem right there :-)

> are to be regenerated, we've to p4-edit them before running the
> generation program. And it's possible to leave a file out (in this case

You can have the generator opening the file before generating it.
Or you can have your make target open this file before calling the
generator.

> the generation program fails, because the file is readonly), or to
> p4-edit a file that won't be modified. I like the way svn figures
> out by itself the set of files it has to commit.

But that requires statting everything. That's fine for a small project.
It's not fine for a large project. CVS doing the same thing was killing
us two years ago, and the project has doubled in size since then --
without p4, it wouldn't have been manageable.

Note that Perforce CAN work in the mode where you change wherever you
want and then it figures out what to do. At the root, "p4 edit ...".
Then, before check-in, "p4 revert -n" to revert anything that didn't
change. However, this mode reduces perforce to the sloth of CVS.

> > Note that the "edit" operation COULD be done off-line in Subversion,
> > because you already have the file, and SVN likes to keep client-side
> > state (Perforce doesn't). However, when on-line, it would be useful if
> > you got a warning if the file you're editing is newer in the repository.
>
> svn status -u ?
> or should the hypothetical "svn edit" command have a -u switch to update
> the file before marking it ready for edition ?

I think it doesn't have to. It would be useful to just warn you if the
file is out of date, but that warning can only actually be implemented if
there is an online repository to check against.

> > This is, in my opinion, one of the better features of Perforce
> (together
> > with bunched atomicly commited change lists).
>
> I don't see how the perforce changelists are better than svn's ones (but
> you're probably referring to a p4 command I don't know.) With p4 you

The feature that perforce will let me commit in 0.1 seconds instead of
crawling my disk for 10 minutes trying to figure out what's open is very
very nice. The feature that perforce will just send me the updates I need
when I sync, without having to crawl the disk to figure out where I last
updated each directory is very nice (although this could be done in SVN
too).

To put it another way: the keeping of state telling the system which files
are actually open (and the general status of each file) in a place where
getting at it is "instantaneous" accelerates working on large code bases
by order(s) of magnitude. Perforce does this server-side. I think that most
of the benefits could be had by doing it client-side, too, which would fit
better in the SVN model.

Cheers,

				/ h+


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Eliminating the text-base penalty

Posted by Rafael Garcia-Suarez <ra...@hexaflux.com>.
"Jon Watte" <hp...@mindcontrol.org> wrote:
> I actually don't care too much about the "extra" copy of each checked 
> out file (disk is cheap!). However, I like working in a command-to-edit 
> mode, because it allows the source control system to know what it needs 
> to inspect for "diff" or "checkin" operations -- only actually edited 
> files are actually considered; no stat-ing of 18,000 files just to 
> figure out what to do.

Yes, for performance reasons that could be useful.

However -- I use Perforce on a project where some generated files
are also kept in the repository. That's means that when those files
are to be regenerated, we've to p4-edit them before running the
generation program. And it's possible to leave a file out (in this case
the generation program fails, because the file is readonly), or to
p4-edit a file that won't be modified. I like the way svn figures
out by itself the set of files it has to commit.

> Note that the "edit" operation COULD be done off-line in Subversion, 
> because you already have the file, and SVN likes to keep client-side 
> state (Perforce doesn't). However, when on-line, it would be useful if 
> you got a warning if the file you're editing is newer in the repository.

svn status -u ?
or should the hypothetical "svn edit" command have a -u switch to update
the file before marking it ready for edition ?

> This is, in my opinion, one of the better features of Perforce (together 
> with bunched atomicly commited change lists).

I don't see how the perforce changelists are better than svn's ones (but
you're probably referring to a p4 command I don't know.) With p4 you
got the changelist for your whole client view. "svn log" gives you the
change log for a file or a subdirectory as well.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

RE: Eliminating the text-base penalty

Posted by Jon Watte <hp...@mindcontrol.org>.
> Option 2 is to implement read-only working copies.  The idea here is
> that, given the appropriate checkout option, files are checked out

My two cents:

I actually don't care too much about the "extra" copy of each checked 
out file (disk is cheap!). However, I like working in a command-to-edit 
mode, because it allows the source control system to know what it needs 
to inspect for "diff" or "checkin" operations -- only actually edited 
files are actually considered; no stat-ing of 18,000 files just to 
figure out what to do.

Note that the "edit" operation COULD be done off-line in Subversion, 
because you already have the file, and SVN likes to keep client-side 
state (Perforce doesn't). However, when on-line, it would be useful if 
you got a warning if the file you're editing is newer in the repository.

This is, in my opinion, one of the better features of Perforce (together 
with bunched atomicly commited change lists). If you could do BOTH in a 
new too, why, I'd save $150/year/person :-)

Cheers,

			/ h+


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Eliminating the text-base penalty

Posted by Philip Martin <ph...@codematters.co.uk>.
"Barry Scott" <ba...@ntlworld.com> writes:

> The size of the object code dwarfs the source size.

Not if your repository contains html pages, or Word documents.

> Optimising a couple of percent is not interesting.

-- 
Philip Martin

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

RE: Eliminating the text-base penalty

Posted by Barry Scott <ba...@ntlworld.com>.

Re: Eliminating the text-base penalty

Posted by David Mankin <ma...@ants.com>.
On Tuesday, September 24, 2002, at 10:59  PM, Greg Hudson wrote:

> Here's a brain dump on getting rid of the text-base penalty, since
> some people expressed interest.
>
> Option 1 is to, given the appropriate checkout option, omit the text
> base and instead store a checksum of the working copy file
...
> Option 2 is to implement read-only working copies.  The idea here is
> that, given the appropriate checkout option, files are checked out
> read-only with no text-base.  When you want to edit a file, you have
> to do an "svn edit filename" to make it read-write; this creates a
> text-base (either by pulling it from the repository or by
> de-substituting it from the working copy; the latter option doesn't
> require the network but would require stashing some notes about what
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> substitutions we performed on each file).  So you only pay a text-base
   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> penalty on files you've chosen to edit.

You mean, like, a "diff"? :-)

I like both options, actually, and can see times when each would be 
valuable separately (and maybe together?).  Option 2 has benefits you 
mentioned, Greg.  Option 1 enables the "save disk space by using 
ra_local", as Florin mentioned.

-David


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Eliminating the text-base penalty

Posted by Branko Čibej <br...@xbc.nu>.
cmpilato@collab.net wrote:

><joking>
>Hack it.  Make the repos config exist as properties on revision 0's root dir.
></joking>
>  
>
You know, that's not such a bad idea at all. Except that I wouldn't make 
them the root dir's props, I'd make them revision props for rev 0 
instead -- those aren't versioned, and can already be modified at any 
time. And all the infrastructure for reading and writing them is already 
there.

-- 
Brane Čibej   <br...@xbc.nu>   http://www.xbc.nu/brane/


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Eliminating the text-base penalty

Posted by cm...@collab.net.
Garrett Rooney <ro...@electricjellyfish.net> writes:

> Kean Johnston wrote:
> 
> > In order to achieve this (and I really don't mind contributing
> > coding cycles to make this happen) I will need to know if there is an
> > appropriate place to store tree-wide parameters that are set either
> > at creation time, or modified after the fact. Some sort of internal
> > svn attributes database. Although I don't know this, I am guessing
> > that there must be a file somewhere whose properties we can use
> > for this purpose.
> 
> at the moment, there is not.  there also needs to be a way for the
> server to push that data down to the clients.  we've wanted this for a
> while, so we can implement project specific keyword expansion (i.e. a
> $FreeBSD$ or $NetBSD$ style $Id$ tag), but nobody has stepped up to do
> the work.
> 
> basically, the repos just needs a config file like the client uses,
> and the RA layers need a way to send that config info back to the
> client.

<joking>
Hack it.  Make the repos config exist as properties on revision 0's root dir.
</joking>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Eliminating the text-base penalty

Posted by Garrett Rooney <ro...@electricjellyfish.net>.
Kean Johnston wrote:

> In order to achieve this (and I really don't mind contributing
> coding cycles to make this happen) I will need to know if there is an
> appropriate place to store tree-wide parameters that are set either
> at creation time, or modified after the fact. Some sort of internal
> svn attributes database. Although I don't know this, I am guessing
> that there must be a file somewhere whose properties we can use
> for this purpose. 

at the moment, there is not.  there also needs to be a way for the 
server to push that data down to the clients.  we've wanted this for a 
while, so we can implement project specific keyword expansion (i.e. a 
$FreeBSD$ or $NetBSD$ style $Id$ tag), but nobody has stepped up to do 
the work.

basically, the repos just needs a config file like the client uses, and 
the RA layers need a way to send that config info back to the client.

-garrett


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

RE: Eliminating the text-base penalty

Posted by Kean Johnston <jk...@caldera.com>.
> Option 1 is to, given the appropriate checkout option, omit the text
> base and instead store a checksum of the working copy file (after
This is my favoured approach. With one slight modification. I think it
should be possible for the repository manager to be able to make this
decision for the users, or at the very least, set the default. What
I'd like to be able to do is some sort of svnadmin command that sets
the policy for all my developers. If an individual developer knows
they will be working offline, then they can over-ride the default
with an svn checkout option. It works in the other direction as well.
If the default is set to USE local cached stored, then the developer
should be able to tell the checkout not to.

In order to achieve this (and I really don't mind contributing
coding cycles to make this happen) I will need to know if there is an
appropriate place to store tree-wide parameters that are set either
at creation time, or modified after the fact. Some sort of internal
svn attributes database. Although I don't know this, I am guessing
that there must be a file somewhere whose properties we can use
for this purpose. However, I think that this should be a setting that
reflects the decision *TODAY*. IE the setting is NOT revision
controlled. I think it has little meaning to apply a decision that may
have been valid 5 years ago, when you pull the tree from 5 years
ago today.

Kean


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org