Re: upgrading source code in local repository

[Ulrich Eckhardt <ul...@dominolaser.com>]

On Friday 29 July 2011, Brecht Ameije wrote:
> I read the referred page in the book, indeed: the script svn_load_dirs.pl
> is exactly what I need. It implements the steps that I did by hand.

There is one thing that took me a while to understand when starting to use 
that: This doesn't preserve your local changes. Instead, after uploading a new 
upstream version, you have to apply your local changes again. Merging helps 
though.


> But when I try it, I doesn't work as flawless as I thought it would:
> It dumps a list with all added/deleted files and gives each of them a
> number.
> Than you have to manually connect the correct numbers to say which ones
> are actually renamed files. A very tedious job, as there are +100 different
> files and the lists aren't even sorted alphabetically :(

I agree. If you know Perl, you might come up with something different/better 
or just more suitable for your case.


> Running /usr/bin/svn add -N --targets
> /tmp/svn_load_dirs_W1GIx88wmF/targets.00001
> /usr/bin/svn_load_dirs.pl: /usr/bin/svn add -N --targets
> /tmp/svn_load_dirs_W1GIx88wmF/targets.00001 failed with this output:
> svn: warning: 'TODO_unicode@' not found
> svn: warning: 'include/ar.h@' not found

The at sign has a special meaning in SVN URL, it is used for peg revisions. 
Question is, are there files with a trailing at sign in the tree that you want 
to import? If yes, they are not handled correctly. If not, wrong filenames 
have been generated at some point. It is probably a bug either in the import 
script or SVN itself.


> Cleaning up /tmp/svn_load_dirs_W1GIx88wmF
> 	(in cleanup) cannot chdir to
> /tmp/svn_load_dirs_W1GIx88wmF/my_import_wc from
> /tmp/svn_load_dirs_W1GIx88wmF: No such file or directory, aborting. at
> /usr/bin/svn_load_dirs.pl line 2056

This looks strange. I know I had issues with virus scanners during imports, 
but I never managed to fully explain why and when.


> The 'not found' files, are actually the files that are added in the new
> vendor version.

Those with the trailing "@"?


> The 'no such file or dir' is in one of the two renamed folders
> ('archival/libarchive') that I marked for renaming...
> Seem like the script is looking for them in my working copy, but they don't
> exist (yet).

The script is checking out a working copy, then copying the version to be 
imported over it. For any missing/added files, it then prompts you if those 
were new file or if they were moved etc. It then performs the according "svn 
add/move/delete" calls and commits the new version. It shouldn't be looking 
for anything in already existing working copies - did you perhaps run it 
inside a working copy?


Uli

-- 
ML: http://subversion.apache.org/docs/community-guide/mailing-lists.html
FAQ: http://subversion.apache.org/faq.html
Docs: http://svnbook.red-bean.com/

**************************************************************************************
Domino Laser GmbH, Fangdieckstraße 75a, 22547 Hamburg, Deutschland
Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932
**************************************************************************************
Visit our website at http://www.dominolaser.com
**************************************************************************************
Diese E-Mail einschließlich sämtlicher Anhänge ist nur für den Adressaten bestimmt und kann vertrauliche Informationen enthalten. Bitte benachrichtigen Sie den Absender umgehend, falls Sie nicht der beabsichtigte Empfänger sein sollten. Die E-Mail ist in diesem Fall zu löschen und darf weder gelesen, weitergeleitet, veröffentlicht oder anderweitig benutzt werden.
E-Mails können durch Dritte gelesen werden und Viren sowie nichtautorisierte Änderungen enthalten. Domino Laser GmbH ist für diese Folgen nicht verantwortlich.
**************************************************************************************

Re: upgrading source code in local repository

[Ulrich Eckhardt <ul...@dominolaser.com>]

On Tuesday 09 August 2011, Andreas Krey wrote:
> On Tue, 09 Aug 2011 09:20:17 +0000, Ulrich Eckhardt wrote:
> ...
> 
> > Yes I am. You might question whether it is reasonable to make them here,
> > but it doesn't matter. The point is that you use svn_load_dirs not in
> > order to just mirror the upstream sources locally, you could use a
> > simple shared directory for that. No, you typically want to modify those
> > sources, like adapt them to your company's build system or similar
> > requirements.
> > 
> > Now, these changes must be applied to any upstream release before you can
> > use it and the astonishing part to me was that svn_load_dirs doesn't
> > help you at all with that.
> 
> No, 'svn'[Note: I'm assuming you meant svn_load_dirs here] ist supposed to
> help you with that. You keep a branch that only contains the upstream
> updates, and repeatedly merge that into the branch that you use to
> modify the vendor's sources to suit your needs. The process of integration
> your modifications with a new upstream release is called 'merging', which
> is something svn can do itself.
> 
> What svn can't do is exactly the history-preserving directory mirroring
> *into* subversions, that, and only that, is what svn_load_dirs is for:
> Change a branch to exactly the state an external directory represents.

Yes, I am fully aware of all that, reread what I said: I warned the OP that 
this is something that initially confused me when I first used svn_load_dirs.

The point is that svn_load_dirs is mentioned as a tool that helps maintaining 
vendor branches. A common task in that context is to update your vendor branch 
from a new upstream release. A requirement there is that local changes are 
preserved through the update. svn_load_dirs doesn't help you with that, it 
only changes "a branch to exactly the state an external directory 
represents.". This is something that confused me when years ago I first 
started creating a vendor branch of something. Reading the docs now, they have 
improved in that aspect.


> Also, how do you think svn_load_dirs *should* do what you request?

Wearing a user's hat, the answer is that I don't care. Yes, this is a rather 
naive approach, but why should I care how it works under the hood? I just need 
my changes preserved!

Now, more practically speaking, I synced a directory to vendor-1.1. I then 
made several changes and then sync to 1.2. The obvious approach is to first 
upload the new 1.2 verbatim and then merge the changes made between the two 
syncs one by one, optionally prompting the user. This is effectively what I do 
manually now and there is no reason it couldn't be done automatically.

I'd even claim that this is superior to the documented approach of merging 
1.1:1.2 into a local branch. The problem is that this is one big change that 
often contains bugfixes that can be contained in the local branch. If upstream 
slightly changed the code, like different formatting, comments, etc, applying 
the big change on top will cause conflicts. Selecting the local changes to 
apply on to of the new 1.2 version manually, possibly adjusting changes in 
between has higher chances of success.

That said, I wouldn't say that it is svn_load_dirs that should do all this. 
Firstly, it is a tool with a very clear goal (sync a directory) and adding 
further features would be just bloat. Secondly, these are just merges and 
those should be easy using _any_ SVN client, so improvements are better put 
there than adding specializations in yet another tool.

> In your scenario it works on a sandbox that contains the last vendor
> release along with your changes on one hand, and it has the current
> vendor release on the other hand. How should it decide what of the
> changes between the two are vendor updates (to be taken from the new
> vendor release) and what are your changes (to be kept)?
> 
> Simply to do that it would not only need to look into the svn history
> of your single branch, but also discern which of the commits were
> for your local adaption and which were vendor updates.

svnsync manages to find out when and where it last synced with, so why 
shouldn't a vendor-branch management tool? You can always add a custom 
property, infer the last sync from the checkin comment or prompt the user.

Cheers!

Uli



-- 
ML: http://subversion.apache.org/docs/community-guide/mailing-lists.html
FAQ: http://subversion.apache.org/faq.html
Docs: http://svnbook.red-bean.com/

**************************************************************************************
Domino Laser GmbH, Fangdieckstraße 75a, 22547 Hamburg, Deutschland
Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932
**************************************************************************************
Visit our website at http://www.dominolaser.com
**************************************************************************************
Diese E-Mail einschließlich sämtlicher Anhänge ist nur für den Adressaten bestimmt und kann vertrauliche Informationen enthalten. Bitte benachrichtigen Sie den Absender umgehend, falls Sie nicht der beabsichtigte Empfänger sein sollten. Die E-Mail ist in diesem Fall zu löschen und darf weder gelesen, weitergeleitet, veröffentlicht oder anderweitig benutzt werden.
E-Mails können durch Dritte gelesen werden und Viren sowie nichtautorisierte Änderungen enthalten. Domino Laser GmbH ist für diese Folgen nicht verantwortlich.
**************************************************************************************

Re: upgrading source code in local repository

[Andreas Krey <a....@gmx.de>]

On Tue, 09 Aug 2011 09:20:17 +0000, Ulrich Eckhardt wrote:
...
> 
> Yes I am. You might question whether it is reasonable to make them here, but 
> it doesn't matter. The point is that you use svn_load_dirs not in order to 
> just mirror the upstream sources locally, you could use a simple shared 
> directory for that. No, you typically want to modify those sources, like adapt 
> them to your company's build system or similar requirements.
> 
> Now, these changes must be applied to any upstream release before you can use 
> it and the astonishing part to me was that svn_load_dirs doesn't help you at 
> all with that.

No, 'svn' ist supposed to help you with that. You keep a branch that only
contains the upstream updates, and repeatedly merge that into the branch
that you use to modify the vendor's sources to suit your needs. The
process of integration your modifications with a new upstream release
is called 'merging', which is something svn can do itself.

What svn can't do is exactly the history-preserving directory mirroring
*into* subversions, that, and only that, is what svn_load_dirs is for:
Change a branch to exactly the state an external directory represents.

Also, how do you think svn_load_dirs *should* do what you request? In
your scenario it works on a sandbox that contains the last vendor
release along with your changes on one hand, and it has the current
vendor release on the other hand. How should it decide what of the
changes between the two are vendor updates (to be taken from the new
vendor release) and what are your changes (to be kept)?

Simply to do that it would not only need to look into the svn history
of your single branch, but also discern which of the commits were
for your local adaption and which were vendor updates.

Andreas

-- 
"Totally trivial. Famous last words."
From: Linus Torvalds <torvalds@*.org>
Date: Fri, 22 Jan 2010 07:29:21 -0800

Re: upgrading source code in local repository

[Ulrich Eckhardt <ul...@dominolaser.com>]

On Wednesday 03 August 2011, you wrote:
> On Aug 1, 2011, at 05:31, Ulrich Eckhardt wrote:
> > On Friday 29 July 2011, Brecht Ameije wrote:
> >> I read the referred page in the book, indeed: the script
> >> svn_load_dirs.pl is exactly what I need. It implements the steps that I
> >> did by hand.
> > 
> > There is one thing that took me a while to understand when starting to
> > use that: This doesn't preserve your local changes. Instead, after
> > uploading a new upstream version, you have to apply your local changes
> > again. Merging helps though.
> 
> I think perhaps you still misunderstand. Correct, svn_load_dirs.pl does not
> preserve your local changes. That's not its job. You never run
> svn_load_dirs.pl on directories or parts of the repository that contain
> any changes. You only run them on pristine copies of the vendor software.
> 
> Example:
> 
> svn mkdir $URL/vendor
> 
> /vendor in the repository will contain copies of all the third-party
> software you want to keep track of. You will never make your own changes
> to that code in this part of the repository. These will always represent
> exact copies of the upstream source.
> 
> Let's say you want to keep track of a third-party software called foo. It's
> currently at version 1.1. You download the tarball of version 1.1 and you
> import it into the repository at this path:
> 
> svn mkdir $URL/vendor/foo
> svn import /path/to/foo-1.1 $URL/vendor/foo/current
> 
> "current" is analogous to the "trunk" of a project that you're developing,
> but it's not called "trunk" here because "trunk" is a place where you make
> changes, and you're not going to be making any changes here.

Yes I am. You might question whether it is reasonable to make them here, but 
it doesn't matter. The point is that you use svn_load_dirs not in order to 
just mirror the upstream sources locally, you could use a simple shared 
directory for that. No, you typically want to modify those sources, like adapt 
them to your company's build system or similar requirements.

Now, these changes must be applied to any upstream release before you can use 
it and the astonishing part to me was that svn_load_dirs doesn't help you at 
all with that. So if you (like we do here) modify "current", you have to 
manually re-apply all those changes after you uploaded a new version. It's no 
big deal, but I at first expected svn_load_dirs to help me with that.


> Now you make a copy of current and call it 1.1:
> 
> svn cp $URL/vendor/foo/current $URL/vendor/foo/1.1
> 
> This is like a "tag" in a project that you're developing.

svn_load_dirs already makes a tag for you, if you want.


> Finally, you copy version 1.1 into another location in the repository where
> you want to use it. Let's say that foo will be used inside a project
> you're developing called bar, so perhaps you run:
> 
> svn cp $URL/vendor/foo/1.1 $URL/bar/trunk/libs/foo

Ahem, no? :-) I don't copy things around just in order to use them. Libraries 
are simply installed and are then available for any project that needs them. 
This restricts changes to be made in just one place. But of course, you can do 
it like that if you want to, I just wanted to point out that there is no One 
True Way(tm).


Cheers!

Uli

-- 
ML: http://subversion.apache.org/docs/community-guide/mailing-lists.html
FAQ: http://subversion.apache.org/faq.html
Docs: http://svnbook.red-bean.com/

**************************************************************************************
Domino Laser GmbH, Fangdieckstraße 75a, 22547 Hamburg, Deutschland
Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932
**************************************************************************************
Visit our website at http://www.dominolaser.com
**************************************************************************************
Diese E-Mail einschließlich sämtlicher Anhänge ist nur für den Adressaten bestimmt und kann vertrauliche Informationen enthalten. Bitte benachrichtigen Sie den Absender umgehend, falls Sie nicht der beabsichtigte Empfänger sein sollten. Die E-Mail ist in diesem Fall zu löschen und darf weder gelesen, weitergeleitet, veröffentlicht oder anderweitig benutzt werden.
E-Mails können durch Dritte gelesen werden und Viren sowie nichtautorisierte Änderungen enthalten. Domino Laser GmbH ist für diese Folgen nicht verantwortlich.
**************************************************************************************

Re: upgrading source code in local repository

[Ryan Schmidt <su...@ryandesign.com>]

On Aug 4, 2011, at 18:11, Michael Diers wrote:

> On 2011-08-03 06:10, Ryan Schmidt wrote:
> [...]
>> I think perhaps you still misunderstand. Correct, svn_load_dirs.pl
>> does not preserve your local changes. That's not its job. You never run
>> svn_load_dirs.pl on directories or parts of the repository that contain
>> any changes. You only run them on pristine copies of the vendor software.
> [...]
> 
> Ryan,
> 
> that is very well put. I suppose the original documentation of
> svn_load_dirs could do with this sort of explanation. Have you
> considered submitting a patch?

I haven't... it's been a long time since I read the book, and I've been on the mailing list many years, so I've forgotten where I originally got the information.

Re: upgrading source code in local repository

[Michael Diers <md...@elego.de>]

On 2011-08-03 06:10, Ryan Schmidt wrote:
[...]
> I think perhaps you still misunderstand. Correct, svn_load_dirs.pl
> does not preserve your local changes. That's not its job. You never run
> svn_load_dirs.pl on directories or parts of the repository that contain
> any changes. You only run them on pristine copies of the vendor software.
[...]

Ryan,

that is very well put. I suppose the original documentation of
svn_load_dirs could do with this sort of explanation. Have you
considered submitting a patch?

-- 
Michael Diers, elego Software Solutions GmbH, http://www.elego.de

Re: upgrading source code in local repository

[Ryan Schmidt <su...@ryandesign.com>]

On Aug 1, 2011, at 05:31, Ulrich Eckhardt wrote:

> On Friday 29 July 2011, Brecht Ameije wrote:
>> I read the referred page in the book, indeed: the script svn_load_dirs.pl
>> is exactly what I need. It implements the steps that I did by hand.
> 
> There is one thing that took me a while to understand when starting to use 
> that: This doesn't preserve your local changes. Instead, after uploading a new 
> upstream version, you have to apply your local changes again. Merging helps 
> though.

I think perhaps you still misunderstand. Correct, svn_load_dirs.pl does not preserve your local changes. That's not its job. You never run svn_load_dirs.pl on directories or parts of the repository that contain any changes. You only run them on pristine copies of the vendor software.

Example:

svn mkdir $URL/vendor

/vendor in the repository will contain copies of all the third-party software you want to keep track of. You will never make your own changes to that code in this part of the repository. These will always represent exact copies of the upstream source.

Let's say you want to keep track of a third-party software called foo. It's currently at version 1.1. You download the tarball of version 1.1 and you import it into the repository at this path:

svn mkdir $URL/vendor/foo
svn import /path/to/foo-1.1 $URL/vendor/foo/current

"current" is analogous to the "trunk" of a project that you're developing, but it's not called "trunk" here because "trunk" is a place where you make changes, and you're not going to be making any changes here.

Now you make a copy of current and call it 1.1:

svn cp $URL/vendor/foo/current $URL/vendor/foo/1.1

This is like a "tag" in a project that you're developing.

Finally, you copy version 1.1 into another location in the repository where you want to use it. Let's say that foo will be used inside a project you're developing called bar, so perhaps you run:

svn cp $URL/vendor/foo/1.1 $URL/bar/trunk/libs/foo

Now you have a copy of foo 1.1 in the libs directory in the trunk of your bar project. You can edit that copy of foo to your heart's content.

Next, imagine a new version 1.2 of foo is released and you want to upgrade the foo in the bar project to that verison.

So you download the tarball of foo 1.2 and use svn_load_dirs.pl to update $URL/vendor/foo/current to version 1.2. svn_load_dirs.pl will also copy that to $URL/vendor/foo/1.2 for you if you ask it to.

Now you can go into your working copy of bar/trunk/libs/foo and run:

svn merge $URL/vendor/foo/1.1 $URL/vendor/foo/1.2 .

That is: You ask subversion to apply changes that occurred between foo 1.1 and 1.2 to your existing (and possibly modified) copy of foo 1.1. You commit it, stating in the message that you updated it to 1.2. Done.


>> Running /usr/bin/svn add -N --targets
>> /tmp/svn_load_dirs_W1GIx88wmF/targets.00001
>> /usr/bin/svn_load_dirs.pl: /usr/bin/svn add -N --targets
>> /tmp/svn_load_dirs_W1GIx88wmF/targets.00001 failed with this output:
>> svn: warning: 'TODO_unicode@' not found
>> svn: warning: 'include/ar.h@' not found
> 
> The at sign has a special meaning in SVN URL, it is used for peg revisions. 
> Question is, are there files with a trailing at sign in the tree that you want 
> to import? If yes, they are not handled correctly. If not, wrong filenames 
> have been generated at some point. It is probably a bug either in the import 
> script or SVN itself.
> 
> 
>> The 'not found' files, are actually the files that are added in the new
>> vendor version.
> 
> Those with the trailing "@"?


Correct, the @ sign has special meaning in subversion. If a filename has an @ in it, subversion will think you are asking for that special meaning and will probably not do what you meant. To work around that, you append an @ to the end of the filename. That is what the script is probably doing here and why you are seeing "@" at the end of each filename. It is not a bug; it is a feature of subversion, and a feature of svn_load_dirs.pl that it is trying to handle all filenames correctly for you, even those containing an @.

This does not explain why it failed for you though.