You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by John Peacock <jp...@rowman.com> on 2005/04/29 14:09:22 UTC
Pre-Design Discussion of Server->client configuration
I was happy that Karl suggested dealing with issue #1974 and friends as
part of the push for 1.3, especially if the design lends itself
to "custom keywords" and possibly the "inherited properties" feature.
As an introduction to that, I thought I'd document some hard-won
knowledge of how the WC code is structured (barring a rewrite of course)
at a very high level. Specifically, I'm going to mostly describe how
keywords work, but I think the behaviors are transferable to
server-resident config issues. I'm sure the more experienced developers
already "know" this, but I want to lay it out so everyone is aware of
the limitations.
1) Subversion is intended to work offline most of the time - by this I
mean that the WC is intended to be largely independent of any
communication with the server except for specific operations that
require it. In terms of keyword expansion, this means that the only
information which can be used to expand keywords must be available in
the working copy files. In particular, it must always be possible to
`svn revert` at any time and recreate the file(s) using only the
information present in the WC itself.
2) The WC is completely modular - by this I mean that any subtree of a
checked out WC can be treated as a firstclass WC in its own right
(including moving the subtree in the filesystem). What this means is
that there is no "top level" or "root" of a working copy. All of the WC
code basically works on the files in a single directory at a time and
then, any subdirectories are recursively handled (unless forbidden with -N).
3) The recursive nature of the WC means that every subtree must have
_all_ of the information available all the time. This either means that
things like inherited properties or server config files have to be
stored either in the WC admin files in each and every subtree or they
have to be stored in some common space outside of the WC tree entirely
(like the .subversion directory). Common config files could be stored
in a hash by UUID in the .subversion directory for caching purposes, but
it would mean that deleting a WC would not remove this information from
the disk (unless a new 'svn release' command were added).
4) Any custom config information (whether talking about server config
overrides or custom keywords) has to be available to the client code
/before/ it begins communicating with the server. In the case of custom
keywords, the working copy file needs to unexpand the changed files
before sending the changes to the server. Similarly, in the case of a
custom log message template, the client code needs that to generate the
log message before sending the commit. Consequently, the
server-mediated config files must be stored locally to the client and
need to be versioned (so the server can see if the client has the most
recent revision, just like the out-of-date check for commits).
I don't know if this is in any way helpful; each of the above is largely
a variation on the same theme.
John
--
John Peacock
Director of Information Research and Technology
Rowman & Littlefield Publishing Group
4501 Forbes Boulevard
Suite H
Lanham, MD 20706
301-459-3366 x.5010
fax 301-429-5748
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Pre-Design Discussion of Server->client configuration
Posted by SteveKing <st...@gmx.ch>.
John Peacock wrote:
> SteveKing wrote:
>
>> - yes, working copies can be used by more than one user. But why would
>> that be a problem for server configs? I mean those configs are
>> intended to pass configurations/other information from the server to
>> *all* working copies (not just individual users). So a working copy
>> used by several people shouldn't be a problem?
>
>
> Greg is saying that _I_ checked out the WC, and got the server config
> files in my .subversion directory. Then _you_ accessed the shared WC
> and did something that requires the server config files, but you never
> got them. It's not a critical error because in many cases, the server
Ah, now I get it. Sometimes I'm a little slow ;)
[snip]
> If the server config included a custom keyword, the client cannot
> perform the revert correctly without having that information (since the
> unexpand/expand would need to know about the new keyword). There are
> probably other cases where the revert requires having the server config
> information in order to restore the file to the original form.
Ok, then a problem with 'revert' would only apply to the also to be
discussed feature 'custom keywords'.
Thanks for explaining!
Stefan
--
___
oo // \\ "De Chelonian Mobile"
(_,\/ \_/ \ TortoiseSVN
\ \_/_\_/> The coolest Interface to (Sub)Version Control
/_/ \_\ http://tortoisesvn.tigris.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Pre-Design Discussion of Server->client configuration
Posted by Branko Čibej <br...@xbc.nu>.
John Peacock wrote:
> If the server config included a custom keyword, the client cannot
> perform the revert correctly without having that information (since
> the unexpand/expand would need to know about the new keyword). There
> are probably other cases where the revert requires having the server
> config information in order to restore the file to the original form.
[a bit off-topic here, but] This is only the case if we allow multiple
spellings of custom keywords; i.e., if you're allowed to put KeywordFoo
in svn:keywords, bu spell it as $Foo$ in the file itself. If we don't
allow that, then the information in svn:keywords is always sufficient
for unexpanding a file.
-- Brane
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Pre-Design Discussion of Server->client configuration
Posted by John Peacock <jp...@rowman.com>.
SteveKing wrote:
> - yes, working copies can be used by more than one user. But why would
> that be a problem for server configs? I mean those configs are intended
> to pass configurations/other information from the server to *all*
> working copies (not just individual users). So a working copy used by
> several people shouldn't be a problem?
Greg is saying that _I_ checked out the WC, and got the server config
files in my .subversion directory. Then _you_ accessed the shared WC
and did something that requires the server config files, but you never
got them. It's not a critical error because in many cases, the server
could signal that your user was not using the latest server config
files, so your commit would fail with an out of date error (and you
could 'svn up' to get the server config in your .subversion directory).
The problem case is that with revert, you couldn't do this (since
revert specifically doesn't require contacting the server).
> by the user perhaps), and a 'revert' would therefore not make much
> sense. I think of such server configs like changes you get when you do
> an 'svn update' - and you can'r 'svn revert' such changes either. You'd
> have to do a merge to get rid of such changes.
If the server config included a custom keyword, the client cannot
perform the revert correctly without having that information (since the
unexpand/expand would need to know about the new keyword). There are
probably other cases where the revert requires having the server config
information in order to restore the file to the original form.
John
--
John Peacock
Director of Information Research and Technology
Rowman & Littlefield Publishing Group
4501 Forbes Boulevard
Suite H
Lanham, MD 20706
301-459-3366 x.5010
fax 301-429-5748
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Pre-Design Discussion of Server->client configuration
Posted by SteveKing <st...@gmx.ch>.
Greg Hudson wrote:
> On Fri, 2005-04-29 at 13:11, steveking wrote:
>
>>Is the '.subversion' directory you mention here the one where the also
>>the 'config' and 'server' file is located? (%APPDATA%\Subversion on
>>Windows)?
>>If yes, then I think that would be the best way to go with server-side
>>configs. We could store them in a subfolder 'serverconfig' like e.g. the
>>authentication data is stored now.
>
>
> Working copies can be used by more than one user. It would be a bug
> (perhaps not a terribly serious one) if you couldn't "svn revert" a home
> directory without access to the repository because you weren't the one
> to check it out.
>
> Also, we should not be assuming that server configuration comes in the
> form of a repository config file. Personally, I think all
> server-to-client configuration goals can be accomplished by adding new
> directory properties.
Seems I have some problems understanding you guys here, so please don't
take this the wrong way.
- yes, working copies can be used by more than one user. But why would
that be a problem for server configs? I mean those configs are intended
to pass configurations/other information from the server to *all*
working copies (not just individual users). So a working copy used by
several people shouldn't be a problem?
- you can't do an 'svn revert' on a working copy and get rid of the
cached auth data either. That's never bothered anyone AFAIK. Also,
server configs are intended to be 'dictated' by the server (overridable
by the user perhaps), and a 'revert' would therefore not make much
sense. I think of such server configs like changes you get when you do
an 'svn update' - and you can'r 'svn revert' such changes either. You'd
have to do a merge to get rid of such changes.
Stefan
--
___
oo // \\ "De Chelonian Mobile"
(_,\/ \_/ \ TortoiseSVN
\ \_/_\_/> The coolest Interface to (Sub)Version Control
/_/ \_\ http://tortoisesvn.tigris.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Pre-Design Discussion of Server->client configuration
Posted by John Peacock <jp...@rowman.com>.
Peter N. Lundblad wrote:
> Depending on what we use server configs for, that might not be desirable.
> It might make operations that today are client-side only to require server
> connectivity.
I think we can reasonably assume that the server-side config data will be slow
changing (compared to any reasonable project churn), and in many cases will be
basically static once the WC is checked out. In the case mentioned, a shared
WC, each user accessing the WC would need to contact the server *once* if they
didn't have the shared data. And again each time the server config data
changed. However the normal condition would be that the WC would have a
complete set of server config files, so client-side only operations (like
revert) will continue to be client-side only.
> I am not sure central config storage is worth it. A few kilobytes for a
> directory shouldn't be a problem. And it allows us to have finer
> granularity than a repository (say you have different project in the same
> repository).
That's really an implementation detail, as to how to handle inherited properties
in the tree. If we want to allow multiple sets of server-side config files to
reside in the tree, having a central storage scheme doesn't prevent it (just
store the config in a hash based on directory location in the tree).
John
--
John Peacock
Director of Information Research and Technology
Rowman & Littlefield Publishing Group
4720 Boston Way
Lanham, MD 20706
301-459-3366 x.5010
fax 301-429-5747
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Pre-Design Discussion of Server->client configuration
Posted by Michael Sinz <mi...@gmail.com>.
On 4/30/05, Peter N. Lundblad <pe...@famlundblad.se> wrote:
> On Sat, 30 Apr 2005, John Peacock wrote:
[...]
> > shared WC case, the client code can "know" that it needs to contact the server
> > to get the server config files when they are not already present in .subversion,
> > yet are flagged in the WC.
> >
> Depending on what we use server configs for, that might not be desirable.
> It might make operations that today are client-side only to require server
> connectivity.
>
> I am not sure central config storage is worth it. A few kilobytes for a
> directory shouldn't be a problem. And it allows us to have finer
> granularity than a repository (say you have different project in the same
> repository).
Hmmm... I think that a "few kilobytes" is not really is direct issue.
However, those "few kilobytes" do multiply up in more ways than one.
First there is the whole directory thing - how many different directories
are in a repository? A number of projects I work with have 5,000+
directories/subdirectories within the repository. A few kilobytes
times 5,000 is starting to add up.
But disk space is "cheap" however disk I/O time is not. Here is
where having files that really are all shared being stored all over
the place starts to make things much slower in multiple ways.
1) You need to read each file - that is at least 3 operations:
open/read/close and most likely more.
2) The OS will need to keep track of more inodes and will try
to cache more filesystem or disk blocks. This puts more preasure
on the cache and thus pushes out data from the cache that is
potentially more useful (assuming that a shared area would work
better)
3) More chance for strange edge cases.
Now, the benefit of having configuration settings being
more fine-grained may be worth the costs. But lets not forget
that it is more than just a "few kilobytes" but that there are
other overheads.
--
Michael Sinz Technology and Engineering Director/Consultant
My place on the web http://www.sinz.org/Michael.Sinz
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Pre-Design Discussion of Server->client configuration
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
On Sun, 1 May 2005, [UTF-8] Branko �^Libej wrote:
> Peter N. Lundblad wrote:
>
> >This means that working copies are no longer self-contained. I think that
> >has been a design goal (but someone could correct me if I'm wrong).
> >
> >
> The features we have on the roadmap right now will require serious
> changes in the working copy model. I would suggest that *no* aspect of
> the current WC model be considered "holy" at this point, including the
> one about working copies being self-contained.
>
OK, I didn't say that feature is especially important to me, just that it
has the implication of changing this design decision.
Maybe it is a bad idea to "redesign the WC" sort of without the complete
rewrite.
Regards,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Pre-Design Discussion of Server->client configuration
Posted by Branko Čibej <br...@xbc.nu>.
Peter N. Lundblad wrote:
>This means that working copies are no longer self-contained. I think that
>has been a design goal (but someone could correct me if I'm wrong).
>
>
The features we have on the roadmap right now will require serious
changes in the working copy model. I would suggest that *no* aspect of
the current WC model be considered "holy" at this point, including the
one about working copies being self-contained.
-- Brane
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Pre-Design Discussion of Server->client configuration
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
On Sat, 30 Apr 2005, John Peacock wrote:
> John Peacock wrote:
> >> Also, we should not be assuming that server configuration comes in the
> >> form of a repository config file. Personally, I think all
> >> server-to-client configuration goals can be accomplished by adding new
> >> directory properties.
> >
> >
> > That's the other option I originally discussed in my #3, but it means
> > that each and every directory in the WC must contain a duplicate of the
> > available server config data. We aren't usually talking about a lot of
> > data, but it is a consideration. It's more of a problem with inherited
> > properties (this time assuming that those are associated with directory
> > properties).
>
> Actually, it occurred to me that we could have both distributed directory
> properties _and_ centralized storage. This gets deeper into implementation than
> we should be when discussing design, but if we mark the WC admin files such that
> "this directory has additional properties" and then /store/ the shared
> properties in the .subversion directory, we get the best of both worlds. The
> directories can share the single set of server-mediated config files, and in the
This means that working copies are no longer self-contained. I think that
has been a design goal (but someone could correct me if I'm wrong).
> shared WC case, the client code can "know" that it needs to contact the server
> to get the server config files when they are not already present in .subversion,
> yet are flagged in the WC.
>
Depending on what we use server configs for, that might not be desirable.
It might make operations that today are client-side only to require server
connectivity.
I am not sure central config storage is worth it. A few kilobytes for a
directory shouldn't be a problem. And it allows us to have finer
granularity than a repository (say you have different project in the same
repository).
Best,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Pre-Design Discussion of Server->client configuration
Posted by John Peacock <jp...@rowman.com>.
John Peacock wrote:
>> Also, we should not be assuming that server configuration comes in the
>> form of a repository config file. Personally, I think all
>> server-to-client configuration goals can be accomplished by adding new
>> directory properties.
>
>
> That's the other option I originally discussed in my #3, but it means
> that each and every directory in the WC must contain a duplicate of the
> available server config data. We aren't usually talking about a lot of
> data, but it is a consideration. It's more of a problem with inherited
> properties (this time assuming that those are associated with directory
> properties).
Actually, it occurred to me that we could have both distributed directory
properties _and_ centralized storage. This gets deeper into implementation than
we should be when discussing design, but if we mark the WC admin files such that
"this directory has additional properties" and then /store/ the shared
properties in the .subversion directory, we get the best of both worlds. The
directories can share the single set of server-mediated config files, and in the
shared WC case, the client code can "know" that it needs to contact the server
to get the server config files when they are not already present in .subversion,
yet are flagged in the WC.
John
--
John Peacock
Director of Information Research and Technology
Rowman & Littlefield Publishing Group
4720 Boston Way
Lanham, MD 20706
301-459-3366 x.5010
fax 301-429-5747
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Pre-Design Discussion of Server->client configuration
Posted by John Peacock <jp...@rowman.com>.
Greg Hudson wrote:
> Working copies can be used by more than one user. It would be a bug
> (perhaps not a terribly serious one) if you couldn't "svn revert" a home
> directory without access to the repository because you weren't the one
> to check it out.
I hadn't thought of that one (I shy away from shared WC's because of the
problems with CVS).
> Also, we should not be assuming that server configuration comes in the
> form of a repository config file. Personally, I think all
> server-to-client configuration goals can be accomplished by adding new
> directory properties.
That's the other option I originally discussed in my #3, but it means
that each and every directory in the WC must contain a duplicate of the
available server config data. We aren't usually talking about a lot of
data, but it is a consideration. It's more of a problem with inherited
properties (this time assuming that those are associated with directory
properties).
John
--
John Peacock
Director of Information Research and Technology
Rowman & Littlefield Publishing Group
4501 Forbes Boulevard
Suite H
Lanham, MD 20706
301-459-3366 x.5010
fax 301-429-5748
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Pre-Design Discussion of Server->client configuration
Posted by Greg Hudson <gh...@MIT.EDU>.
On Fri, 2005-04-29 at 13:11, steveking wrote:
> Is the '.subversion' directory you mention here the one where the also
> the 'config' and 'server' file is located? (%APPDATA%\Subversion on
> Windows)?
> If yes, then I think that would be the best way to go with server-side
> configs. We could store them in a subfolder 'serverconfig' like e.g. the
> authentication data is stored now.
Working copies can be used by more than one user. It would be a bug
(perhaps not a terribly serious one) if you couldn't "svn revert" a home
directory without access to the repository because you weren't the one
to check it out.
Also, we should not be assuming that server configuration comes in the
form of a repository config file. Personally, I think all
server-to-client configuration goals can be accomplished by adding new
directory properties.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Pre-Design Discussion of Server->client configuration
Posted by SteveKing <st...@gmx.ch>.
John Peacock wrote:
> steveking wrote:
>
>> John Peacock wrote:
>>
>>> Common config files could be stored in a hash by UUID in the
>>> .subversion directory for caching purposes, but it would mean that
>>> deleting a WC would not remove this information from the disk (unless
>>> a new 'svn release' command were added).
>>
>> Is the '.subversion' directory you mention here the one where the also
>> the 'config' and 'server' file is located? (%APPDATA%\Subversion on
>> Windows)?
>
> Yes, sorry, I should have been more specific (or more general, depending
> on how you look at it). The one [possibly serious] drawback is that
> this data isn't "connected" to the WC in any way except internal to the
> client code, so deleting the WC files in the filesystem doesn't remove
> all data associated with that repository.
Why would that be a serious drawback? I mean the same applies to the
auth data cached right now, and I've never heard of a drawback (except
for that data not being encrypted, but that's another issue and
something which is taken care of in 1.2 on windows).
Can you provide maybe some examples why this would be a drawback?
Stefan
--
___
oo // \\ "De Chelonian Mobile"
(_,\/ \_/ \ TortoiseSVN
\ \_/_\_/> The coolest Interface to (Sub)Version Control
/_/ \_\ http://tortoisesvn.tigris.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Pre-Design Discussion of Server->client configuration
Posted by John Peacock <jp...@rowman.com>.
steveking wrote:
> John Peacock wrote:
>> Common config files could
>> be stored in a hash by UUID in the .subversion directory for caching
>> purposes, but it would mean that deleting a WC would not remove this
>> information from the disk (unless a new 'svn release' command were
>> added).
>
>
> Is the '.subversion' directory you mention here the one where the also
> the 'config' and 'server' file is located? (%APPDATA%\Subversion on
> Windows)?
Yes, sorry, I should have been more specific (or more general, depending
on how you look at it). The one [possibly serious] drawback is that
this data isn't "connected" to the WC in any way except internal to the
client code, so deleting the WC files in the filesystem doesn't remove
all data associated with that repository.
And a simple 'svn release' isn't trivial, since you could have multiple
WC's from the same repository (so you wouldn't want to delete those
files). I suppose the repository could just push out the missing data
if ever the config files were deleted (on purpose or by accident).
John
--
John Peacock
Director of Information Research and Technology
Rowman & Littlefield Publishing Group
4501 Forbes Boulevard
Suite H
Lanham, MD 20706
301-459-3366 x.5010
fax 301-429-5748
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Pre-Design Discussion of Server->client configuration
Posted by steveking <st...@gmx.ch>.
John Peacock wrote:
[snip]
> 3) The recursive nature of the WC means that every subtree must have
> _all_ of the information available all the time. This either means that
> things like inherited properties or server config files have to be
> stored either in the WC admin files in each and every subtree or they
> have to be stored in some common space outside of the WC tree entirely
> (like the .subversion directory). Common config files could be stored
> in a hash by UUID in the .subversion directory for caching purposes, but
> it would mean that deleting a WC would not remove this information from
> the disk (unless a new 'svn release' command were added).
Is the '.subversion' directory you mention here the one where the also
the 'config' and 'server' file is located? (%APPDATA%\Subversion on
Windows)?
If yes, then I think that would be the best way to go with server-side
configs. We could store them in a subfolder 'serverconfig' like e.g. the
authentication data is stored now.
Stefan
--
___
oo // \\ "De Chelonian Mobile"
(_,\/ \_/ \ TortoiseSVN
\ \_/_\_/> The coolest Interface to (Sub)Version Control
/_/ \_\ http://tortoisesvn.tigris.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org