You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Bo Berglund <bo...@gmail.com> on 2018/01/11 11:43:28 UTC
EOL problems in cvs2svn converted repository
I have converted our CVS repositories to SVN using cvs2svn.py v 2.5.0.
It looked like it was successful but now when I am working in the
checked out code I noticed that the line endings are UNIX style even
when checking out on Windows (our main working platform).
With CVS we always got a conversion from UNIX LF (in the repository)
to Windows CRLF in the working copy.
Not so now...
I have read the SVNBook 1.7 about EOL sequences but I do not
understand what I can do to fix this.
It will not be possible to go through every text file in the
repository and change its "properties" because we store such an
immense number of files in our repositories.
Is there any way to tell the svn client to set line endings to Windows
style when checking out on Windows (similar to the timestamp option I
asked about recently)? And return to repository style on commit...
I found a section in the conf file named [auto-props] but it seems
only to be applicable for commit and import, not for checkout AFAICT.
And all of that section is commented out by default.
--
Bo Berglund
Developer in Sweden
Re: EOL problems in cvs2svn converted repository
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Johan Corveleyn wrote on Fri, 09 Feb 2018 13:40 +0100:
> So changing this in historic revisions (in a dump file) is not just a
> question of adding the property, it also involves normalizing the
> corresponding file contents in the dump (and the dump format contains
> checksums and offsets, so manipulating actual contents in not easy).
Why wouldn't it be easy?
If the dumpstream was generated without --deltas, the records are not
dependent on each other, so writing some sort of stdin-to-stdout filter
that renormalizes newlines of some file records would be
straightforward. It wouldn't be doable with sed, for the reasons you
give, but it'd be doable using the API.
If the dumpstream was generated with --deltas, it would be easier to load
the dumpstream into a repository and then to use svnsync to renormalize
the newlines.
(Conceptually, 'svnsync sync' should allow the source to be a
dumpstream, reusing the dumpstream-to-editor backend of 'svnrdump load',
but that hasn't been implemented.)
Cheers,
Daniel
Re: EOL problems in cvs2svn converted repository
Posted by Ryan Schmidt <su...@ryandesign.com>.
On Feb 9, 2018, at 14:48, Johan Corveleyn wrote:
> ISTR there used to be a helper script to do this (to apply a certain
> autoprops configuration to your working copy ...
> svn-apply-autoprops.py or .pl or something), but I can't find it.
> Maybe it was in the contrib folder which has been removed since (you
> might be able to find a version of it googling).
https://svn.apache.org/repos/asf/subversion/trunk/contrib/client-side/svn_apply_autoprops.py
Re: EOL problems in cvs2svn converted repository
Posted by Johan Corveleyn <jc...@gmail.com>.
On Fri, Feb 9, 2018 at 8:04 PM, Bo Berglund <bo...@gmail.com> wrote:
> On Fri, 9 Feb 2018 13:40:43 +0100, Johan Corveleyn <jc...@gmail.com>
> wrote:
>
>>> We use VisualSVN with svn version 1.9.7 on the server
>>
>>Standard dump/load cannot do this.
>>
>>However, you might be able to do it using this tool:
>>https://github.com/jwiegley/svndumptool
>>
>>(it can manipulate dump files by setting properties and stuff like
>>that, and performing various eol fixes)
>>
>>I'm wondering how robust this is though ... make sure to doublecheck
>>the result (for instance by checking out / exporting some of these
>>files from different points in history).
>>
>>The reason why I'm wondering is that, internally (in the repository) a
>>text file is stored in "normalized form" (which is LF-terminated) if
>>you have the svn:eol-style=native property set. But if it's not set,
>>svn doesn't alter the incoming file in any way (it could be binary for
>>all svn knows), so if that file was committed from Windows, it will
>>probably be in CRLF format, so that's what will be stored in the
>>repository. So changing this in historic revisions (in a dump file) is
>>not just a question of adding the property, it also involves
>>normalizing the corresponding file contents in the dump (and the dump
>>format contains checksums and offsets, so manipulating actual contents
>>in not easy).
>
> OK, I think I will be OK with only setting the trunk version files
> correctly, if possible. After all this is what the consultants will
> use.
> And the development IDE:s seem to be able to cope fine with UNIX style
> source files (at least those I have tested with).
>
> Someone suggested that this would do it, but I am not sure:
>
> svn propset svn:eol-style 'native' -R *
>
> And he did not say how to do it, for example if I have to do that
> inside a working copy or if it is possible directly against the
> repository.
You'll need a working copy for that.
> In any case it seems like a dangerous proposition using * as file
> spec, since there are binary files also in the repo (icons, images,
> Windows executables, dll's etc).
>
> So maybe a script like this (executed at the top of a working copy):
>
> svn propset svn:eol-style 'native' -R *.txt
> svn propset svn:eol-style 'native' -R *.c
> svn propset svn:eol-style 'native' -R *.h
> svn propset svn:eol-style 'native' -R *.pas
> svn propset svn:eol-style 'native' -R *.dpr
> svn propset svn:eol-style 'native' -R *.ini
> svn propset svn:eol-style 'native' -R *.conf
> svn propset svn:eol-style 'native' -R *.py
> svn propset svn:eol-style 'native' -R *.cpp
> svn propset svn:eol-style 'native' -R *.lpr
> svn propset svn:eol-style 'native' -R *.lpi
> svn propset svn:eol-style 'native' -R *.bat
> svn propset svn:eol-style 'native' -R *.iss
> svn propset svn:eol-style 'native' -R *.xml
> svn propset svn:eol-style 'native' -R *.gld
> svn propset svn:eol-style 'native' -R *.inc
> svn propset svn:eol-style 'native' -R *.dsp
> svn propset svn:eol-style 'native' -R *.dsw
I don't think that will work. The *.txt will only be interpreted by
your shell, and it will not be passed to the recursive propset command
... so it'll only apply the propset to *.txt files in the current
directory. Perhaps you can whip something up in combination with the
find command:
find . -name '*.txt' -exec svn propset svn:eol-style native {} \;
or something like that (note: you don't need to put quotes around the
value, otherwise the value will actually become 'native', instead of
native).
> And then a svn commit
>
> Would this work?
> What would happen if a file already has the requested property?
I think that will work. Not sure what happens if the file already has
the property ... but it's easy to try it out, so just give it a try
:-).
ISTR there used to be a helper script to do this (to apply a certain
autoprops configuration to your working copy ...
svn-apply-autoprops.py or .pl or something), but I can't find it.
Maybe it was in the contrib folder which has been removed since (you
might be able to find a version of it googling).
--
Johan
Re: EOL problems in cvs2svn converted repository
Posted by Ryan Schmidt <su...@ryandesign.com>.
On Feb 9, 2018, at 13:04, Bo Berglund wrote:
> OK, I think I will be OK with only setting the trunk version files
> correctly, if possible. After all this is what the consultants will
> use.
> And the development IDE:s seem to be able to cope fine with UNIX style
> source files (at least those I have tested with).
>
> Someone suggested that this would do it, but I am not sure:
>
> svn propset svn:eol-style 'native' -R *
>
> And he did not say how to do it, for example if I have to do that
> inside a working copy or if it is possible directly against the
> repository.
Like most nontrivial changes, you stage the changes in a working copy, examine the result and make sure everything looks ok, then you commit.
> In any case it seems like a dangerous proposition using * as file
> spec, since there are binary files also in the repo (icons, images,
> Windows executables, dll's etc).
Right, if you have binary files in the working copy, you don't want Subversion to corrupt them by performing eol translation on them, so you should not use "*"; you should be more selective about what types of files you apply the property to.
> So maybe a script like this (executed at the top of a working copy):
>
> svn propset svn:eol-style 'native' -R *.txt
> svn propset svn:eol-style 'native' -R *.c
> svn propset svn:eol-style 'native' -R *.h
> svn propset svn:eol-style 'native' -R *.pas
> svn propset svn:eol-style 'native' -R *.dpr
> svn propset svn:eol-style 'native' -R *.ini
> svn propset svn:eol-style 'native' -R *.conf
> svn propset svn:eol-style 'native' -R *.py
> svn propset svn:eol-style 'native' -R *.cpp
> svn propset svn:eol-style 'native' -R *.lpr
> svn propset svn:eol-style 'native' -R *.lpi
> svn propset svn:eol-style 'native' -R *.bat
> svn propset svn:eol-style 'native' -R *.iss
> svn propset svn:eol-style 'native' -R *.xml
> svn propset svn:eol-style 'native' -R *.gld
> svn propset svn:eol-style 'native' -R *.inc
> svn propset svn:eol-style 'native' -R *.dsp
> svn propset svn:eol-style 'native' -R *.dsw
>
> And then a svn commit
>
> Would this work?
> What would happen if a file already has the requested property?
Whether or not the file had the "svn:eol-style" property to begin with, running "svn propset svn:eol-style native" on it will set the "svn:eol-style" property to the value "native".
Your above proposal won't work, unless you don't have subdirectories. This is because your shell, not Subversion, expands globs like "*.txt", and only in the current directory.
For example, if your directory contains:
a.txt
b.txt
c/
c/d.txt
Then "svn propset svn:eol-style 'native' -R *.txt" will be expanded by the shell to "svn propset svn:eol-style 'native' -R a.txt b.txt" before Subversion ever sees the command. Then Subversion will set the property of a.txt and b.txt. The "-R" flag means "recursively process directories" but you didn't specify any directories, so no recursive directory processing occurs.
What you may want to do instead is use the "find" command to recursively locate the files you want to modify. (I'm presuming you're running these commands on a UNIX-like operating system; I don't know what the equivalent Windows commands might be.) For example, in your working copy:
find -E . -iregex '^.*\.(bat|c|conf|cpp|dpr|dsp|dsw|gld|h|inc|ini|iss|lpi|lpr|pas|py|txt|xml)$' -print0 | xargs -0 svn propset svn:eol-style native
Now you can use "svn status" and "svn diff" to inspect the staged changes in the working copy, and either commit them if satisfied, or "svn revert -R ." if you want to discard the changes.
This presumes that you can identify all of the files by extension. If your repository users use Windows, this might be the case. But on other operating systems, it's not unusual to find files with names like "README", with no ".txt" extension, which are nevertheless text files that you might want to apply the svn:eol-style property to. In that case, you may have to use a criterion other than extension to identify your files. For example, you could inspect the MIME type, and modify all files whose MIME type begins with "text/":
find . -type f | file --mime-type --no-pad --separator :: --files-from - | sed -n 's,:: text/.*$,,p' | tr '\n' '\0' | xargs -0 svn propset svn:eol-style native
Re: EOL problems in cvs2svn converted repository
Posted by Bo Berglund <bo...@gmail.com>.
On Fri, 9 Feb 2018 13:40:43 +0100, Johan Corveleyn <jc...@gmail.com>
wrote:
>> We use VisualSVN with svn version 1.9.7 on the server
>
>Standard dump/load cannot do this.
>
>However, you might be able to do it using this tool:
>https://github.com/jwiegley/svndumptool
>
>(it can manipulate dump files by setting properties and stuff like
>that, and performing various eol fixes)
>
>I'm wondering how robust this is though ... make sure to doublecheck
>the result (for instance by checking out / exporting some of these
>files from different points in history).
>
>The reason why I'm wondering is that, internally (in the repository) a
>text file is stored in "normalized form" (which is LF-terminated) if
>you have the svn:eol-style=native property set. But if it's not set,
>svn doesn't alter the incoming file in any way (it could be binary for
>all svn knows), so if that file was committed from Windows, it will
>probably be in CRLF format, so that's what will be stored in the
>repository. So changing this in historic revisions (in a dump file) is
>not just a question of adding the property, it also involves
>normalizing the corresponding file contents in the dump (and the dump
>format contains checksums and offsets, so manipulating actual contents
>in not easy).
OK, I think I will be OK with only setting the trunk version files
correctly, if possible. After all this is what the consultants will
use.
And the development IDE:s seem to be able to cope fine with UNIX style
source files (at least those I have tested with).
Someone suggested that this would do it, but I am not sure:
svn propset svn:eol-style 'native' -R *
And he did not say how to do it, for example if I have to do that
inside a working copy or if it is possible directly against the
repository.
In any case it seems like a dangerous proposition using * as file
spec, since there are binary files also in the repo (icons, images,
Windows executables, dll's etc).
So maybe a script like this (executed at the top of a working copy):
svn propset svn:eol-style 'native' -R *.txt
svn propset svn:eol-style 'native' -R *.c
svn propset svn:eol-style 'native' -R *.h
svn propset svn:eol-style 'native' -R *.pas
svn propset svn:eol-style 'native' -R *.dpr
svn propset svn:eol-style 'native' -R *.ini
svn propset svn:eol-style 'native' -R *.conf
svn propset svn:eol-style 'native' -R *.py
svn propset svn:eol-style 'native' -R *.cpp
svn propset svn:eol-style 'native' -R *.lpr
svn propset svn:eol-style 'native' -R *.lpi
svn propset svn:eol-style 'native' -R *.bat
svn propset svn:eol-style 'native' -R *.iss
svn propset svn:eol-style 'native' -R *.xml
svn propset svn:eol-style 'native' -R *.gld
svn propset svn:eol-style 'native' -R *.inc
svn propset svn:eol-style 'native' -R *.dsp
svn propset svn:eol-style 'native' -R *.dsw
And then a svn commit
Would this work?
What would happen if a file already has the requested property?
--
Bo Berglund
Developer in Sweden
Re: EOL problems in cvs2svn converted repository
Posted by Johan Corveleyn <jc...@gmail.com>.
On Fri, Feb 9, 2018 at 11:44 AM, Bo Berglund <bo...@gmail.com> wrote:
> On Thu, 11 Jan 2018 09:51:54 -0600, Ryan Schmidt
> <su...@ryandesign.com> wrote:
>
>>> I found a section in the conf file named [auto-props] but it seems
>>> only to be applicable for commit and import, not for checkout AFAICT.
>>> And all of that section is commented out by default.
>>
>>If you want Subversion to perform EOL translation on a file, you must set
>>the svn:eol-style property on that file to the desired value.
>>There is no other way.
>
> Really? I imagine that a lot of people have been bit by this and
> someone must have come up with a way to fix the problem without having
> to manually go over all of the files in a repository matching the file
> type (by extension) and set the property.
> Is there no script or such available?
>
>>
>>If you want all files with a particular name or extension to get this
>>property when they are added to a working copy, set that up in the
>>auto-props section of your Subversion client's file.
>>This will apply to that client only and would need to be repeated for
>>all clients / all users.
>
> I have put a Readme.txt file into the repo that describes the way to
> set up the svn client side conf file.
> But it will help only for new files...
>
>>
>>If using Subversion client 1.8 or later, use a repository-dictated
>>auto-props configuration:
>>
>>https://subversion.apache.org/docs/release-notes/1.8.html#repos-dictated-config
>>
>>Auto-props only cause the property to be added to the file when the file
>>is added to the working copy. A user can subvert auto-props, and a user
>>can also delete the property after it was added, before or after the file
>>was committed.
>>
>>If you want to enforce that this property is used on files that need it,
>>and/or prevent users from removing it, write a pre-commit hook that does that.
>
> But this only affects files added to the repo, right?
> I need something that can apply to *all* existing files of the
> affected types.
>
>>If you want all of the files you're importing from CVS that match the criteria
>>to have the property, adjust your cvs-to-svn conversion process to arrange for
>>that to be the case.
>>
>
> This will be a hurdle since the conversion is already done and the
> repository is in use...
> But *how* would one do this if a new conversion has to be done?
> I am not fluent in Python, so any such coding is beyond me...
>
> There is one repository holding a number of projects prepared for use
> by an external contractor, which has so far not been used for
> development. So in theory I can repeat the conversion cvs2svn for this
> with some cvs2svn settings tweak that will set the svn:eol-style
> conversion correctly.
> But I need to know how this can be done in that case.
>
> Is it possible to use dump/load?
> --------------------------------
> Is it possible to somehow edit an svn dump file (with a script) to
> look for all the target files and add the eol-style property setting
> for each file so it will be effected in the svn load?
> When I look at a dumnp file load it looks like what it does is
> automatically performing all the commit operations that was done in
> the source repository....
>
> Or maybe simpler:
> -----------------
> Create dump file of the repo.
> Create a new repo and set its automatic property settings as discussed
> above.
> Then load the dump file.
> Will this make the new repository contain the automatically set eol
> properties? I.e. will the load be affected by the global repository
> automatic property settings?
>
> Note that our repos are each containing a number of "projects" in the
> top level of the repo and all of these have the structure of a svn
> repo with trunk, branches and tags subdirectories.
>
> We use VisualSVN with svn version 1.9.7 on the server
Standard dump/load cannot do this.
However, you might be able to do it using this tool:
https://github.com/jwiegley/svndumptool
(it can manipulate dump files by setting properties and stuff like
that, and performing various eol fixes)
I'm wondering how robust this is though ... make sure to doublecheck
the result (for instance by checking out / exporting some of these
files from different points in history).
The reason why I'm wondering is that, internally (in the repository) a
text file is stored in "normalized form" (which is LF-terminated) if
you have the svn:eol-style=native property set. But if it's not set,
svn doesn't alter the incoming file in any way (it could be binary for
all svn knows), so if that file was committed from Windows, it will
probably be in CRLF format, so that's what will be stored in the
repository. So changing this in historic revisions (in a dump file) is
not just a question of adding the property, it also involves
normalizing the corresponding file contents in the dump (and the dump
format contains checksums and offsets, so manipulating actual contents
in not easy).
--
Johan
Re: EOL problems in cvs2svn converted repository
Posted by Ryan Schmidt <su...@ryandesign.com>.
On Feb 11, 2018, at 10:49, Bo Berglund wrote:
> On Sun, 11 Feb 2018 08:45:17 +0000, Daniel Shahaf wrote:
>
>> Bo Berglund wrote on Sun, 11 Feb 2018 09:34 +0100:
>>> For OpenVPN I found that I had to add some servers to apt-get in order
>>> to use the OpenVPN own repository, is there a similar solution for
>>> Subversion?
>>
>> See https://subversion.apache.org/packages
>>
>> ... but unless there is a specific bugfix from 1.9.4..1.9.7 that you
>> need, 1.9.3 should be fine. See
>> https://svn.apache.org/repos/asf/subversion/trunk/CHANGES
>> (that file includes information for all past releases and some future ones)
>
> Thanks,
> I went ahead and left the installed version at 1.9.3 on the Ubuntu
> 16.04 machine. Installed OpenVPN on this and could connect to the
> server LAN using my usual OVPN file.
>
> With this I could check out a few small projects to test on and I
> found that the svn_apply_autoprops.py ran just fine on this.
> And if it can do what it is meant to do it seems like a good tool to
> use!
>
> However I have now a few projects that have been modified and I wanted
> to check what came out, so inside the top folder of each modified
> project I ran svn diff <filename> to see what had happened and it
> turns out that it looks kind of strange to me....
>
> For <filename>.txt:
>
> svn diff Readme.txt
> Index: Readme.txt
> ===================================================================
> --- Readme.txt (revision 16)
> +++ Readme.txt (working copy)
>
> Property changes on: Readme.txt
> ___________________________________________________________________
> Added: svn:keywords
> ## -0,0 +1 ##
> +Author Date Id Rev URL
>
>
> Here I cannot see that the native EOL I want added is applied...
I agree, the property was not set there by running svn_apply_autoprops.py. Maybe that's because it was already set before? Run "svn proplist -v Readme.txt" to see the properties that are currently set on the file.
> Instead some property "Keywords" which I lnow nothing about is being
> set.
Your autoprops settings (the default ones that you uncommented) instruct Subversion to set those keywords to be expanded in text files. If you don't want to use keyword expansion, don't instruct Subversion to do that (i.e. remove those entries from autoprops). Keyword substitution is explained in the book:
http://svnbook.red-bean.com/en/1.7/svn.advanced.props.special.keywords.html
> For other Pascal files (*.pas, *.dpr etc):
>
> svn diff BuildModel.pas
> Index: BuildModel.pas
> ===================================================================
> --- BuildModel.pas (revision 16)
> +++ BuildModel.pas (working copy)
>
> Property changes on: BuildModel.pas
> ___________________________________________________________________
> Added: svn:eol-style
> ## -0,0 +1 ##
> +native
> \ No newline at end of property
>
>
> In this case I see +native, so I assume it means thet the eol-style is
> modified, but what does the last line mean?
It means that the svn:eol-style property's value does not end with a newline. In other words, the property's value is the 6 characters "native", not the 6 characters "native" followed by the newline character as you might otherwise assume had that message not been printed. It is normal for property values not to end with a newline, so you will normally see this message.
> I have modified the subversion conf file as follows:
>
> [miscellany]
> global-ignores = *.o *.lo *.la *.al .libs *.so *.so.[0-9]* *.a
> *.pyc *.pyo *.a *.aps *.bak.clw *.db *.dbg *.dcu
> *.ddp *.dsk *.dti *.err *.exp *.gid *.gz *.bak
> *.identcache *.ilk *.jar *.lnk *.local *.log *.lps
> *.lrt *.mod *.ncb *.ocx *.opt *.pch
> backup lib dcu
> *.pjt *.plg *.pdb *.ppu *.obj *.hpp
> *.or *.rsj *.rst *.sbr *.scc *.sln *.tar *.tds
> *.tlh *.tli *.trg *.vbw *.zip __history bak *.~* .#*
>
> use-commit-times = yes
> enable-auto-props = yes
>
> [auto-props]
> *.c = svn:eol-style=native
> *.cpp = svn:eol-style=native
> *.h = svn:keywords=Author Date Id Rev URL;svn:eol-style=native
> *.dsp = svn:eol-style=CRLF
> *.dsw = svn:eol-style=CRLF
> *.sh = svn:eol-style=native;svn:executable
> *.txt = svn:eol-style=native;svn:keywords=Author Date Id Rev URL;
> *.png = svn:mime-type=image/png
> *.jpg = svn:mime-type=image/jpeg
> Makefile = svn:eol-style=native
> *.pas = svn:eol-style=native
> *.dpr = svn:eol-style=native
> *.dfm = svn:eol-style=native
> *.lpr = svn:eol-style=native
> *.lpi = svn:eol-style=native
> *.ini = svn:eol-style=native
> *.conf = svn:eol-style=native
> *.bat = svn:eol-style=native
> *.iss = svn:eol-style=native
> *.xml = svn:eol-style=native
> *.gld = svn:eol-style=native
> *.inc = svn:eol-style=native
>
> Note that the auto-props settings from *.c until MakeFile are just
> using the standard settings that were present as defaults in the conf
> file but commented out. I uncommented these.
> The ones below Makefile I have added myself.
>
> So the Readme.txt is affected by this line:
> *.txt = svn:eol-style=native;svn:keywords=Author Date Id Rev URL;
>
> I have no idea what good the second setting on this line really does,
> maybe it should not be there at all???
>
> Any advice?
Re: EOL problems in cvs2svn converted repository
Posted by Bo Berglund <bo...@gmail.com>.
On Sun, 11 Feb 2018 17:49:32 +0100, Bo Berglund
<bo...@gmail.com> wrote:
>Any advice?
Thanks for all advice and links. Most helpful.
I have now mastered the way the svn_apply_autoprops.py can be used and
have completed the EOL configuration of the repository for contractors
with all items therein.
Had to do as follows:
- Used a virtual machine running Ubuntu Server 16.04-3LTS (the py file
would not run on Windows)
- Installed subversion via apt-get, resulted in version 1.9.7
- Edited the ~/.subversion/config file and added all of the file types
that should have native EOL to the autoprops section
- Connected the machine to the remote network using OpenVPN
- Checked out all of the projects in the repository
- Ran the svn_apply_autoprops.py program in the projects
- Committed the changes
- Went back to Windows and updated my working copies and now I have
CRLF line endings.
In the process I have used the svn revert command several times when
the result pointed to a flaw in the config files, very handy!
So I guess I am done now (with this issue)!
--
Bo Berglund
Developer in Sweden
Re: EOL problems in cvs2svn converted repository
Posted by Bo Berglund <bo...@gmail.com>.
On Sun, 11 Feb 2018 08:45:17 +0000, Daniel Shahaf
<d....@daniel.shahaf.name> wrote:
>Bo Berglund wrote on Sun, 11 Feb 2018 09:34 +0100:
>> For OpenVPN I found that I had to add some servers to apt-get in order
>> to use the OpenVPN own repository, is there a similar solution for
>> Subversion?
>
>See https://subversion.apache.org/packages
>
>... but unless there is a specific bugfix from 1.9.4..1.9.7 that you
>need, 1.9.3 should be fine. See
>https://svn.apache.org/repos/asf/subversion/trunk/CHANGES
>(that file includes information for all past releases and some future ones)
Thanks,
I went ahead and left the installed version at 1.9.3 on the Ubuntu
16.04 machine. Installed OpenVPN on this and could connect to the
server LAN using my usual OVPN file.
With this I could check out a few small projects to test on and I
found that the svn_apply_autoprops.py ran just fine on this.
And if it can do what it is meant to do it seems like a good tool to
use!
However I have now a few projects that have been modified and I wanted
to check what came out, so inside the top folder of each modified
project I ran svn diff <filename> to see what had happened and it
turns out that it looks kind of strange to me....
For <filename>.txt:
svn diff Readme.txt
Index: Readme.txt
===================================================================
--- Readme.txt (revision 16)
+++ Readme.txt (working copy)
Property changes on: Readme.txt
___________________________________________________________________
Added: svn:keywords
## -0,0 +1 ##
+Author Date Id Rev URL
Here I cannot see that the native EOL I want added is applied...
Instead some property "Keywords" which I lnow nothing about is being
set.
For other Pascal files (*.pas, *.dpr etc):
svn diff BuildModel.pas
Index: BuildModel.pas
===================================================================
--- BuildModel.pas (revision 16)
+++ BuildModel.pas (working copy)
Property changes on: BuildModel.pas
___________________________________________________________________
Added: svn:eol-style
## -0,0 +1 ##
+native
\ No newline at end of property
In this case I see +native, so I assume it means thet the eol-style is
modified, but what does the last line mean?
I have modified the subversion conf file as follows:
[miscellany]
global-ignores = *.o *.lo *.la *.al .libs *.so *.so.[0-9]* *.a
*.pyc *.pyo *.a *.aps *.bak.clw *.db *.dbg *.dcu
*.ddp *.dsk *.dti *.err *.exp *.gid *.gz *.bak
*.identcache *.ilk *.jar *.lnk *.local *.log *.lps
*.lrt *.mod *.ncb *.ocx *.opt *.pch
backup lib dcu
*.pjt *.plg *.pdb *.ppu *.obj *.hpp
*.or *.rsj *.rst *.sbr *.scc *.sln *.tar *.tds
*.tlh *.tli *.trg *.vbw *.zip __history bak *.~* .#*
use-commit-times = yes
enable-auto-props = yes
[auto-props]
*.c = svn:eol-style=native
*.cpp = svn:eol-style=native
*.h = svn:keywords=Author Date Id Rev URL;svn:eol-style=native
*.dsp = svn:eol-style=CRLF
*.dsw = svn:eol-style=CRLF
*.sh = svn:eol-style=native;svn:executable
*.txt = svn:eol-style=native;svn:keywords=Author Date Id Rev URL;
*.png = svn:mime-type=image/png
*.jpg = svn:mime-type=image/jpeg
Makefile = svn:eol-style=native
*.pas = svn:eol-style=native
*.dpr = svn:eol-style=native
*.dfm = svn:eol-style=native
*.lpr = svn:eol-style=native
*.lpi = svn:eol-style=native
*.ini = svn:eol-style=native
*.conf = svn:eol-style=native
*.bat = svn:eol-style=native
*.iss = svn:eol-style=native
*.xml = svn:eol-style=native
*.gld = svn:eol-style=native
*.inc = svn:eol-style=native
Note that the auto-props settings from *.c until MakeFile are just
using the standard settings that were present as defaults in the conf
file but commented out. I uncommented these.
The ones below Makefile I have added myself.
So the Readme.txt is affected by this line:
*.txt = svn:eol-style=native;svn:keywords=Author Date Id Rev URL;
I have no idea what good the second setting on this line really does,
maybe it should not be there at all???
Any advice?
--
Bo Berglund
Developer in Sweden
Re: EOL problems in cvs2svn converted repository
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Bo Berglund wrote on Sun, 11 Feb 2018 09:34 +0100:
> For OpenVPN I found that I had to add some servers to apt-get in order
> to use the OpenVPN own repository, is there a similar solution for
> Subversion?
See https://subversion.apache.org/packages
... but unless there is a specific bugfix from 1.9.4..1.9.7 that you
need, 1.9.3 should be fine. See
https://svn.apache.org/repos/asf/subversion/trunk/CHANGES
(that file includes information for all past releases and some future ones)
Re: EOL problems in cvs2svn converted repository
Posted by Bo Berglund <bo...@gmail.com>.
On Sun, 11 Feb 2018 08:40:06 +0100, Bo Berglund
<bo...@gmail.com> wrote:
>OK,
>then I will have to test on a Linux machine. The problem here is that
>I need to connect VPN to the server from Linux. So I have more
>research to do for that....
>The Linux machine I have available is an OpenVPN *server* and I have
>tried to figure out how I can connect it as a Client to the remote
>LAN. But it seems like once connected I cannot disconnect the client
>without bringing down the server or rebooting. Very strange.
>But OT here of course.
Now I found a virtual machine I could use, it is an Ubuntu Server
16.04-3LTS built last year.
But subversion on it is only 1.9.3 and if I try apt-get install
subversion it tells me that it is already the latest version, which
indicates that the ubuntu repository lags behind.
Is there a way to update svn to 1.9.7 on Ubuntu 16.04?
For OpenVPN I found that I had to add some servers to apt-get in order
to use the OpenVPN own repository, is there a similar solution for
Subversion?
--
Bo Berglund
Developer in Sweden
Re: EOL problems in cvs2svn converted repository
Posted by Bo Berglund <bo...@gmail.com>.
On Sun, 11 Feb 2018 00:13:27 -0600, Ryan Schmidt
<su...@ryandesign.com> wrote:
>> Maybe it is not supposed to work on Windows?
>> I have Python 2.7.1 installed in my PC.
>> Should I instead check out the projects on Linux to make it work?
>
>According to a quick Google search, Python on Windows does not have
>spawnvp, so the script will not work on Windows as written.
>
OK,
then I will have to test on a Linux machine. The problem here is that
I need to connect VPN to the server from Linux. So I have more
research to do for that....
The Linux machine I have available is an OpenVPN *server* and I have
tried to figure out how I can connect it as a Client to the remote
LAN. But it seems like once connected I cannot disconnect the client
without bringing down the server or rebooting. Very strange.
But OT here of course.
--
Bo Berglund
Developer in Sweden
Re: EOL problems in cvs2svn converted repository
Posted by Ryan Schmidt <su...@ryandesign.com>.
On Feb 10, 2018, at 09:29, Bo Berglund wrote:
> I tried the contrib py script on my Windows machine as follows:
>
> python svn_apply_autoprops.py --config %APPDATA%\Subversion\config
> D:\Engineering\Projects\Subversion\EI
> Traceback (most recent call last):
> File "svn_apply_autoprops.py", line 195, in <module>
> sys.exit(main())
> File "svn_apply_autoprops.py", line 192, in main
> os.path.walk(wc_path, filter_walk, autoprop_lines)
> File "C:\Programs\Python\lib\ntpath.py", line 263, in walk
> walk(name, func, arg)
> File "C:\Programs\Python\lib\ntpath.py", line 259, in walk
> func(arg, top, names)
> File "svn_apply_autoprops.py", line 147, in filter_walk
> status = os.spawnvp(os.P_WAIT, 'svn', command)
> AttributeError: 'module' object has no attribute 'spawnvp'
>
>
> Maybe it is not supposed to work on Windows?
> I have Python 2.7.1 installed in my PC.
> Should I instead check out the projects on Linux to make it work?
According to a quick Google search, Python on Windows does not have spawnvp, so the script will not work on Windows as written.
Re: EOL problems in cvs2svn converted repository
Posted by Bo Berglund <bo...@gmail.com>.
On Fri, 9 Feb 2018 14:06:33 -0600, Ryan Schmidt
<su...@ryandesign.com> wrote:
>If you want to do it after the conversion by just setting the
>property starting now, without altering history, then check out
>a working copy, set the properties where you want them, then check it in.
I think this is what I want to do, but preferrably in a semi-automatic
way. I.e. I would like to automatically find all instances of affected
files (via the client config auto-properties entries) and then apply
to the current working copy. So the contrib Python script
svn_apply_autoprops.py might be a choice.
And, yes, I am working on Windows with this.
I tried the contrib py script on my Windows machine as follows:
python svn_apply_autoprops.py --config %APPDATA%\Subversion\config
D:\Engineering\Projects\Subversion\EI
Traceback (most recent call last):
File "svn_apply_autoprops.py", line 195, in <module>
sys.exit(main())
File "svn_apply_autoprops.py", line 192, in main
os.path.walk(wc_path, filter_walk, autoprop_lines)
File "C:\Programs\Python\lib\ntpath.py", line 263, in walk
walk(name, func, arg)
File "C:\Programs\Python\lib\ntpath.py", line 259, in walk
func(arg, top, names)
File "svn_apply_autoprops.py", line 147, in filter_walk
status = os.spawnvp(os.P_WAIT, 'svn', command)
AttributeError: 'module' object has no attribute 'spawnvp'
Maybe it is not supposed to work on Windows?
I have Python 2.7.1 installed in my PC.
Should I instead check out the projects on Linux to make it work?
--
Bo Berglund
Developer in Sweden
Re: EOL problems in cvs2svn converted repository
Posted by Ryan Schmidt <su...@ryandesign.com>.
On Feb 9, 2018, at 04:44, Bo Berglund wrote:
> On Thu, 11 Jan 2018 09:51:54 -0600, Ryan Schmidt wrote:
>
>>> I found a section in the conf file named [auto-props] but it seems
>>> only to be applicable for commit and import, not for checkout AFAICT.
>>> And all of that section is commented out by default.
>>
>> If you want Subversion to perform EOL translation on a file, you must set
>> the svn:eol-style property on that file to the desired value.
>> There is no other way.
>
> Really? I imagine that a lot of people have been bit by this and
> someone must have come up with a way to fix the problem without having
> to manually go over all of the files in a repository matching the file
> type (by extension) and set the property.
> Is there no script or such available?
What a lot of people were bit by was the way CVS handles eol translation. When applied to files that it should not be applied to, such as binary files, eol translation is destructive and irreversible. Subversion aims to be a better CVS, so Subversion improves on CVS by requiring you to be explicit about which files you want it to perform eol translation on, so that it never performs an eol translation you didn't request.
>> If you want all files with a particular name or extension to get this
>> property when they are added to a working copy, set that up in the
>> auto-props section of your Subversion client's file.
>> This will apply to that client only and would need to be repeated for
>> all clients / all users.
>
> I have put a Readme.txt file into the repo that describes the way to
> set up the svn client side conf file.
> But it will help only for new files...
Yes.
>> If using Subversion client 1.8 or later, use a repository-dictated
>> auto-props configuration:
>>
>> https://subversion.apache.org/docs/release-notes/1.8.html#repos-dictated-config
>>
>> Auto-props only cause the property to be added to the file when the file
>> is added to the working copy. A user can subvert auto-props, and a user
>> can also delete the property after it was added, before or after the file
>> was committed.
>>
>> If you want to enforce that this property is used on files that need it,
>> and/or prevent users from removing it, write a pre-commit hook that does that.
>
> But this only affects files added to the repo, right?
Yes.
> I need something that can apply to *all* existing files of the
> affected types.
You need both.
>> If you want all of the files you're importing from CVS that match the criteria
>> to have the property, adjust your cvs-to-svn conversion process to arrange for
>> that to be the case.
>>
>
> This will be a hurdle since the conversion is already done and the
> repository is in use...
> But *how* would one do this if a new conversion has to be done?
> I am not fluent in Python, so any such coding is beyond me...
>
> There is one repository holding a number of projects prepared for use
> by an external contractor, which has so far not been used for
> development. So in theory I can repeat the conversion cvs2svn for this
> with some cvs2svn settings tweak that will set the svn:eol-style
> conversion correctly.
> But I need to know how this can be done in that case.
>
> Is it possible to use dump/load?
> --------------------------------
> Is it possible to somehow edit an svn dump file (with a script) to
> look for all the target files and add the eol-style property setting
> for each file so it will be effected in the svn load?
> When I look at a dumnp file load it looks like what it does is
> automatically performing all the commit operations that was done in
> the source repository....
>
> Or maybe simpler:
> -----------------
> Create dump file of the repo.
> Create a new repo and set its automatic property settings as discussed
> above.
> Then load the dump file.
> Will this make the new repository contain the automatically set eol
> properties? I.e. will the load be affected by the global repository
> automatic property settings?
>
> Note that our repos are each containing a number of "projects" in the
> top level of the repo and all of these have the structure of a svn
> repo with trunk, branches and tags subdirectories.
If you want to set the svn:eol-style property during the conversion, you'll have to read the documentation of the conversion tool or scripts you're using; I assume they would provide for this but I don't know.
If you want to do it after the conversion and fix it in all revisions by editing repository history and forcing all users to throw away their old working copies and check out new ones, then you can "svnadmin dump" the repo, then use the third-party svndumptool program to set svn:eol-style where you want it, then "svnadmin load" the modified dumpfile.
If you want to do it after the conversion by just setting the property starting now, without altering history, then check out a working copy, set the properties where you want them, then check it in.
Re: EOL problems in cvs2svn converted repository
Posted by Bo Berglund <bo...@gmail.com>.
On Thu, 11 Jan 2018 09:51:54 -0600, Ryan Schmidt
<su...@ryandesign.com> wrote:
>> I found a section in the conf file named [auto-props] but it seems
>> only to be applicable for commit and import, not for checkout AFAICT.
>> And all of that section is commented out by default.
>
>If you want Subversion to perform EOL translation on a file, you must set
>the svn:eol-style property on that file to the desired value.
>There is no other way.
Really? I imagine that a lot of people have been bit by this and
someone must have come up with a way to fix the problem without having
to manually go over all of the files in a repository matching the file
type (by extension) and set the property.
Is there no script or such available?
>
>If you want all files with a particular name or extension to get this
>property when they are added to a working copy, set that up in the
>auto-props section of your Subversion client's file.
>This will apply to that client only and would need to be repeated for
>all clients / all users.
I have put a Readme.txt file into the repo that describes the way to
set up the svn client side conf file.
But it will help only for new files...
>
>If using Subversion client 1.8 or later, use a repository-dictated
>auto-props configuration:
>
>https://subversion.apache.org/docs/release-notes/1.8.html#repos-dictated-config
>
>Auto-props only cause the property to be added to the file when the file
>is added to the working copy. A user can subvert auto-props, and a user
>can also delete the property after it was added, before or after the file
>was committed.
>
>If you want to enforce that this property is used on files that need it,
>and/or prevent users from removing it, write a pre-commit hook that does that.
But this only affects files added to the repo, right?
I need something that can apply to *all* existing files of the
affected types.
>If you want all of the files you're importing from CVS that match the criteria
>to have the property, adjust your cvs-to-svn conversion process to arrange for
>that to be the case.
>
This will be a hurdle since the conversion is already done and the
repository is in use...
But *how* would one do this if a new conversion has to be done?
I am not fluent in Python, so any such coding is beyond me...
There is one repository holding a number of projects prepared for use
by an external contractor, which has so far not been used for
development. So in theory I can repeat the conversion cvs2svn for this
with some cvs2svn settings tweak that will set the svn:eol-style
conversion correctly.
But I need to know how this can be done in that case.
Is it possible to use dump/load?
--------------------------------
Is it possible to somehow edit an svn dump file (with a script) to
look for all the target files and add the eol-style property setting
for each file so it will be effected in the svn load?
When I look at a dumnp file load it looks like what it does is
automatically performing all the commit operations that was done in
the source repository....
Or maybe simpler:
-----------------
Create dump file of the repo.
Create a new repo and set its automatic property settings as discussed
above.
Then load the dump file.
Will this make the new repository contain the automatically set eol
properties? I.e. will the load be affected by the global repository
automatic property settings?
Note that our repos are each containing a number of "projects" in the
top level of the repo and all of these have the structure of a svn
repo with trunk, branches and tags subdirectories.
We use VisualSVN with svn version 1.9.7 on the server
--
Bo Berglund
Developer in Sweden
Re: EOL problems in cvs2svn converted repository
Posted by Ryan Schmidt <su...@ryandesign.com>.
On Jan 11, 2018, at 05:43, Bo Berglund wrote:
> I have converted our CVS repositories to SVN using cvs2svn.py v 2.5.0.
> It looked like it was successful but now when I am working in the
> checked out code I noticed that the line endings are UNIX style even
> when checking out on Windows (our main working platform).
> With CVS we always got a conversion from UNIX LF (in the repository)
> to Windows CRLF in the working copy.
> Not so now...
>
> I have read the SVNBook 1.7 about EOL sequences but I do not
> understand what I can do to fix this.
> It will not be possible to go through every text file in the
> repository and change its "properties" because we store such an
> immense number of files in our repositories.
> Is there any way to tell the svn client to set line endings to Windows
> style when checking out on Windows (similar to the timestamp option I
> asked about recently)? And return to repository style on commit...
>
> I found a section in the conf file named [auto-props] but it seems
> only to be applicable for commit and import, not for checkout AFAICT.
> And all of that section is commented out by default.
If you want Subversion to perform EOL translation on a file, you must set the svn:eol-style property on that file to the desired value. There is no other way.
If you want all files with a particular name or extension to get this property when they are added to a working copy, set that up in the auto-props section of your Subversion client's file. This will apply to that client only and would need to be repeated for all clients / all users.
If using Subversion client 1.8 or later, use a repository-dictated auto-props configuration:
https://subversion.apache.org/docs/release-notes/1.8.html#repos-dictated-config
Auto-props only cause the property to be added to the file when the file is added to the working copy. A user can subvert auto-props, and a user can also delete the property after it was added, before or after the file was committed.
If you want to enforce that this property is used on files that need it, and/or prevent users from removing it, write a pre-commit hook that does that.
If you want all of the files you're importing from CVS that match the criteria to have the property, adjust your cvs-to-svn conversion process to arrange for that to be the case.