You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Brad Spencer <sp...@jacknife.org> on 2004/03/29 22:21:27 UTC
Modeling recursive CVS modules in subversion-1.0.x
Hello!
(Apologies if I sent an empty message to the list moments ago via some
keyboard mashing.)
We've been considering what would be needed to move a large CVS
repository to subversion-1.0.x, and I think we've wrapped figured out
the right way to model everything we currently do except for one
important detail: recursively defined modules. Let me illustrate the
issue:
We have a lot of smallish libraries within one repository. There are
dependencies between these libraries in the sense that some higher
level library needs one or more lower-level libraries checked out in
order to build. Right now, we model these dependencies with the CVS
modules file. For example, consider the following libraries (one per
'lib*' directory):
general/libOne (stands alone)
general/libTwo (directly depends on libOne)
general/libThree (directly depends on libOne)
general/libFour (directly depends on libTwo and libThree)
app/program1 (directly depends on libOne)
app/program2 (directly depends on libTwo and libFour)
So the CVS modules file is set up with aliases that are essentially
like this:
general_libOne -a general/libOne
general_libTwo -a general/libOne general/libTwo
general_libThree -a general/libOne general/libThree
general_libFour -a general/libTwo general/libThree (no libOne)
app_program1 -a general/libOne app/program1
app_program2 -a general/libTwo general/libFour app/program2
So, if I check out app_program2, I automatically get its code and all
of its dependencies:
general/libOne
general/libTwo
general/libThre
general/libFour
app/program2
There are a large number of these small libraries (think hundreds; we
say "libraries are cheap" in the same way subversion says "copies are
cheap" and we use the same code base for a wide collection of
application domains). The dependencies between them are captured
manually when they are created or changed, but they need to be
automatically followed on checkouts and they need to be versioned on
tags and branches. Ideally, new dependencies would appear on an "svn
update" too, but that's not critical. The working copy needs to be
"unified" so that that I can edit libTwo and program2 as checked out
this way and commit once to save the changes.
At first, I thought svn:externals was going to cover this
functionality, but it seems to focus on external relationships. I
really want to capture internal relationships, which is what the CVS
modules file is for, essentially.
So, the best idea I have come up with so far is to make a file in the
repository that essentially has the same list of relationships as the
CVS modules file that I "svn cat" through a script that builds a list
of what to checkout (i.e. what CVS does internally). Then, I can use
the output of that script to check out all the bits I need. But then
that doesn't make a "unified" working copy that I can commit to (in
fact, I'm not convinced this will even build the directory shape I
need).
Essentially, I have a tree in the repository, and I only want to check
out those paths of the tree (in space, not time) that are dependencies
for what I am working on, but then I want to be able to edit them all
together. In other words, I want to go "svn co http://.../trunk" or
"svn co http://.../branches/branch1" except have it pruned to include
only those paths that I specify in a "relative to base-URI" way.
I'm wondering if there's a way (a "subversion way") to capture
these relationships between my code, or if this problem has come up
before and how it was addressed. Thanks!
--
----------------------------------------------------------------
Brad Spencer - spencer@jacknife.org - "It's quite nice..."
"S.M.A.K.I.B.B.F.B." - A.J. Rimmer | http://jacknife.org/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Modeling recursive CVS modules in subversion-1.0.x
Posted by John Rousseau <JR...@novell.com>.
Me too, me too!
http://www.contactor.se/~dast/svnusers/archive-2004-06/1177.shtml
-John
On Sat, 31 Jul 2004 14:40:33 -0300, Brad Spencer <sp...@jacknife.org> wrote:
> On Sat, Jul 31, 2004 at 12:45:54PM -0300, spencer wrote:
>> (For example http://www.contactor.se/~dast/svnusers/archive-2004-02/0144.shtml
>> seems to be asking the same question in a different way.)
>
> BTW, I believe this would also solve the problems of these other recent
> postings:
>
> http://www.contactor.se/~dast/svnusers/archive-2004-03/0797.shtml
> http://www.contactor.se/~dast/svnusers/archive-2004-07/1202.shtml
> http://www.contactor.se/~dast/svnusers/archive-2004-07/1525.shtml
> http://www.contactor.se/~dast/svnusers/archive-2004-07/1730.shtml
>
--
----------------------------------------------------------------
John Rousseau JRousseau@novell.com
Novell, Inc. Phone: +1 781 464 8377
404 Wyman Street Fax: +1 781 464 8100
Waltham, MA 02451 http://www.novell.com
----------------------------------------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Modeling recursive CVS modules in subversion-1.0.x
Posted by Brad Spencer <sp...@jacknife.org>.
On Sat, Jul 31, 2004 at 12:45:54PM -0300, spencer wrote:
> (For example http://www.contactor.se/~dast/svnusers/archive-2004-02/0144.shtml
> seems to be asking the same question in a different way.)
BTW, I believe this would also solve the problems of these other recent
postings:
http://www.contactor.se/~dast/svnusers/archive-2004-03/0797.shtml
http://www.contactor.se/~dast/svnusers/archive-2004-07/1202.shtml
http://www.contactor.se/~dast/svnusers/archive-2004-07/1525.shtml
http://www.contactor.se/~dast/svnusers/archive-2004-07/1730.shtml
--
----------------------------------------------------------------
Brad Spencer - spencer@jacknife.org - "It's quite nice..."
"S.M.A.K.I.B.B.F.B." - A.J. Rimmer | http://jacknife.org/
Re: Modeling recursive CVS modules in subversion-1.0.x
Posted by Brad Spencer <sp...@jacknife.org>.
I posted this message a few months back and had some discussions on
IRC with Ben Collins-Sussman at the time. We essentially discovered
that there were a few outstanding svn "bugs" that would keep me from
doing what I needed to do. So I figured I'd wait a few months and
check into it again.
I've done some playing around with "version 1.0.5 (r9954)", and I've
searched online and found many others asking essentially the same
question, never with a real answer of how to do this.
(For examplehttp://www.contactor.se/~dast/svnusers/archive-2004-02/0144.shtml
seems to be asking the same question in a different way.)
It's really the only thing holding our group back from being about to
move from CVS to SVN, so I'm trying to figure out how we can answer
it.
I think that all I need to be able to do is to prune a checkout to
specific (relative) paths within the subtree. There are two ways to
do this in CVS:
1. Check out or "update -d" each directory one at a time.
2. Use modules to define the parts of the tree you want and check
that module out "all at once". In CVS, this is just the first
option in disguise, but from a user's point of view it's nice
because you don't have to manually specify the list of
directories.
In subversion, there appears to be no way to do this.
1. Checking everything out and removing the parts you don't need is
not a realistic solution. There is just to much content to do
that, and it means checkouts are O(size of everything) even if
you just need some of it.
2. Checking out each directory one at a time doesn't work because
they become "disjoint working copies" and I can't make changes
throughout and then commit. There is a real danger of losing
work here if your tools don't recursively search all
revision-controlled directories for changes and then commit
them. These kind of changes are frequent.
3. svn:externals doesn't work, because I can't make changes
throughout the checked out content and commit them atomically
(even though they are in the same repository). That, and there
is no notion of a relative reference (I still think an
"svn:internals" that did support relative reference would solve
this problem completely). AFAIK, "svn:externals" are almost
useless for intra-repository references because of these two
reasons.
I guess I just don't understand why something as simple as "start at
this URL (as a kind of base URI) and check out the following relative
paths" can't be done. For example:
$ svn co -b file:///my/repository/trunk . general/libOne \
general/libTwo src
A src/general
A src/general/libOne
A src/general/libOne/file.txt
A src/general/libTwo
A src/general/libTwo/file.txt
$ cd src
$ svn st -v
5 5 general
5 5 general/libOne
5 5 general/libOne/file.txt
5 5 general/libTwo
5 5 general/libTwo/file.txt
$ echo "more text" >> general/libOne/file.txt
$ echo "more text" >> general/libTwo/file.txt
$ svn commit
svn commit -m test
Sending general/libOne/file.txt
Sending general/libTwo/file.txt
Transmitting file data ..
Committed revision 6.
# And I can pull in more directories that I didn't ask for in the
original checkout (as long as they are within the base URI + one of
the pieces I checked out). But since I checked out ".", I can get
everything under the base URI.
$ cd general
$ svn update libThree
A libThree
A libThree/file2.txt
A libThree/file1.txt
Updated to revision 6.
Note that if I change the base URI to a branch, or tag, the _same_ set
of relative URIs will work. If I store those relative URIs in a
version-controlled file themselves (think CVS modules file), I can
even tag and branch the relationships. To do the checkout, I just
"svn cat" the relationships I want and use them as the list of
relative URLs for "svn co". Or at least I could start there and
develop a better scheme later.
My original message follows, for reference. Thanks for your time.
On Mon, Mar 29, 2004 at 06:21:27PM -0400, spencer wrote:
> Hello!
>
> (Apologies if I sent an empty message to the list moments ago via some
> keyboard mashing.)
>
> We've been considering what would be needed to move a large CVS
> repository to subversion-1.0.x, and I think we've wrapped figured out
> the right way to model everything we currently do except for one
> important detail: recursively defined modules. Let me illustrate the
> issue:
>
> We have a lot of smallish libraries within one repository. There are
> dependencies between these libraries in the sense that some higher
> level library needs one or more lower-level libraries checked out in
> order to build. Right now, we model these dependencies with the CVS
> modules file. For example, consider the following libraries (one per
> 'lib*' directory):
>
> general/libOne (stands alone)
> general/libTwo (directly depends on libOne)
> general/libThree (directly depends on libOne)
> general/libFour (directly depends on libTwo and libThree)
>
> app/program1 (directly depends on libOne)
> app/program2 (directly depends on libTwo and libFour)
>
> So the CVS modules file is set up with aliases that are essentially
> like this:
>
> general_libOne -a general/libOne
> general_libTwo -a general/libOne general/libTwo
> general_libThree -a general/libOne general/libThree
> general_libFour -a general/libTwo general/libThree (no libOne)
>
> app_program1 -a general/libOne app/program1
> app_program2 -a general/libTwo general/libFour app/program2
>
> So, if I check out app_program2, I automatically get its code and all
> of its dependencies:
>
> general/libOne
> general/libTwo
> general/libThre
> general/libFour
> app/program2
>
> There are a large number of these small libraries (think hundreds; we
> say "libraries are cheap" in the same way subversion says "copies are
> cheap" and we use the same code base for a wide collection of
> application domains). The dependencies between them are captured
> manually when they are created or changed, but they need to be
> automatically followed on checkouts and they need to be versioned on
> tags and branches. Ideally, new dependencies would appear on an "svn
> update" too, but that's not critical. The working copy needs to be
> "unified" so that that I can edit libTwo and program2 as checked out
> this way and commit once to save the changes.
>
> At first, I thought svn:externals was going to cover this
> functionality, but it seems to focus on external relationships. I
> really want to capture internal relationships, which is what the CVS
> modules file is for, essentially.
>
> So, the best idea I have come up with so far is to make a file in the
> repository that essentially has the same list of relationships as the
> CVS modules file that I "svn cat" through a script that builds a list
> of what to checkout (i.e. what CVS does internally). Then, I can use
> the output of that script to check out all the bits I need. But then
> that doesn't make a "unified" working copy that I can commit to (in
> fact, I'm not convinced this will even build the directory shape I
> need).
>
> Essentially, I have a tree in the repository, and I only want to check
> out those paths of the tree (in space, not time) that are dependencies
> for what I am working on, but then I want to be able to edit them all
> together. In other words, I want to go "svn co http://.../trunk" or
> "svn co http://.../branches/branch1" except have it pruned to include
> only those paths that I specify in a "relative to base-URI" way.
>
> I'm wondering if there's a way (a "subversion way") to capture
> these relationships between my code, or if this problem has come up
> before and how it was addressed. Thanks!
>
> --
> ----------------------------------------------------------------
> Brad Spencer - spencer@jacknife.org - "It's quite nice..."
> "S.M.A.K.I.B.B.F.B." - A.J. Rimmer | http://jacknife.org/
--
----------------------------------------------------------------
Brad Spencer - spencer@jacknife.org - "It's quite nice..."
"S.M.A.K.I.B.B.F.B." - A.J. Rimmer | http://jacknife.org/
Re: Modeling recursive CVS modules in subversion-1.0.x
Posted by Brad Spencer <sp...@infointeractive.com>.
On Mon, Mar 29, 2004 at 06:21:27PM -0400, spencer wrote:
> We have a lot of smallish libraries within one repository. There are
> dependencies between these libraries in the sense that some higher
> level library needs one or more lower-level libraries checked out in
> order to build. Right now, we model these dependencies with the CVS
> modules file. For example, consider the following libraries (one per
> 'lib*' directory):
I while back, I posted the message I'm replying to, available as
http://subversion.tigris.org/servlets/ReadMsg?list=users&msgNo=8940
Later, I chatted with Ben Collins-Sussman about svm:externals and the
like some more. Here are some snippets of this, which I'm sure he
won't mind since it was in the public #svn channel.
<sussman> if you run 'svn up' from the top of your wc, it will update
everything, following the svn:externals list
<sussman> commits won't, and that's an acknowledged bug.
<sussman> it means you'll have to commit a few times.
Except in my case "a few times" is more like "fifty or a hundred". :(
<spencer_> well, if I'm on a branch, then the externals will all point
to the trunk via the absolute URLs
<sussman> spencer_: when you copy /trunk to /branches/foo
<sussman> you'll need to change all the externals properties in the
branch
<sussman> to point to dirs in the branch
<sussman> it's a bit awkward, admittedly.
Upon reflection, too awkward, I'm afraid :(
<spencer_> how do i cleanly add to a current checkout... like the
equiv of "cvs update -d subdir"...
<sussman> there's no way to do that.
<sussman> well, it's a bug, at least.
<sussman> if you do a nonrecursive checkout, then try to 'update' a
missing dir, it doesn't work currently.
<sussman> basically, any use of --nonrecursive is busted.
The equivalent of "cvs update -d subdir" would allow us to simulate it
outside of svn. We could maintain version control on the "module"
relationships by storing the relationship list in svn, just like the
CVS modules file.
<spencer_> no prob really... bugs are better than "not designed that
way" because we are not talking immediate term
<spencer_> we are just starting to plan this
I've also noticed similarities in issues #1336 and, to a lesser
extent, #1167. Are these the issues I should be following in order to
know if/when svn will be feasible for our repository? Thanks for your
help!
--
------------------------------------------------------------------
Brad Spencer - spencer@infointeractive.com - "It's quite nice..."
Systems Architect | InfoInterActive Corp. | A Canadian AOL Company
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org