You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Christian Unger <ch...@mac.com> on 2004/10/07 17:51:11 UTC
Can't recode string
Hi,
I posted this a while ago. is there really no solution for the "Can't
recode string" message ?
[OS: Mac OS X 10.3.5]
[Subversion: version 1.1(metissian)]
French and Japanese filenames lead to commit abort with error "Can't
recode string".
which settings does svn need to be able to recode the strings of those
files ?
with best regards
christian unger
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Can't recode string
Posted by Patrick Smears <pa...@ensoft.co.uk>.
On Fri, 8 Oct 2004, John Williams wrote:
> I found issues #1847 and #1946 on this subject.
> Both of these have been marked invalid, the second merely
> because it was not discussed on this list.
>
> The problem is that svn is not able to handle characters in
> the filename which do not match the current LANG character set,
> or are not valid UTF-8 characters, if LANG is not set.
>
> Issue #1847 asserts that this is not a subversion issue.
> I wish to dispute that.
>
> 1) Subversion claims to be able to handle binary data.
> But this problem shows that it cannot handle binary data
> in the filename, even when the host OS allows it.
The trouble is that filenames are not binary data - it's character data.
The difference is that character data has a different binary
representation depending on what character set is in use. So the character
'lowercase e acute' is '0xe9' in ISO-8859-1, '0x82' in DOS codepage 775,
'0xc3 0xa9' in UTF-8, and doesn't exist at all in ISO-8859-5.
Internally, subversion stores characters in UTF-8, which allows for all
characters defined in the Unicode standards to be represented (which for
all practical purposes means 'every character possible'). However, since
people's terminals etc do not always accept UTF-8, subversion converts the
characters to the appropriate format before output.
This is great, because it means that a user with an ISO-8859-1 terminal,
one with a UTF-8 terminal and one with a CP775 terminal can all be
accessing the same repository, and view the filename containing a
'lowercase e acute' correctly. However, in order to be able to do this,
subversion does need to be told which character set to use - this seems
fair enough (especially as it takes its information from the standard
environment variables for this purpose).
> 2) If a project has filenames in more than one character set,
> I don't think it is the revision control system's place
> to disallow that. A revision control system should be able
> to accept any file, and give it back in exactly the same way.
Agreed - and (once it knows what character set you're using), subversion
allows this. The restriction it does place, however, is that in order to
view a filename, you must be using a character set that supports that
character.
This is perhaps overly strict (in that it can make it hard to find out
what the problem is when there is a file with a troublesome name), but
consistent with subversion's philosophy of 'don't change the data' - it
will not display the filename in a misleading way.
However, you can force the commands to always succeed by setting up your
environment variables to use UTF-8. (Characters other than the ASCII ones
won't display correctly unless your terminal is also set up for UTF-8, but
at least this allows for debugging to take place.)
> 3) This error causes "svn status" to abort, not reporting the
> status of the remaining files.
I agree that this is less than ideal, but it's not immediately obvious to
me what the 'preferred' behaviour should be - how should it display
characters that are not in your current character set? Perhaps some form
of quoting - but that could easily confuse scripts that are parsing the
output...
At the very least, the error message should be improved to be more
helpful.
> 4) This error causes "svn add" to abort, leaving the directory
> in some sort of half-baked state, which svn revert will not revert.
That's bad... perhaps you could post a transcript of how to reproduce this
problem, and the errors it causes, so that people on the list can
investigate? Include the settings of your environment variables (LANG and
LC*) as these will affect the way subversion operates...
Patrick
--
The easy way to type accents in Windows: http://www.frkeys.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Can't recode string
Posted by John Williams <wi...@tni.com>.
I found issues #1847 and #1946 on this subject.
Both of these have been marked invalid, the second merely
because it was not discussed on this list.
The problem is that svn is not able to handle characters in
the filename which do not match the current LANG character set,
or are not valid UTF-8 characters, if LANG is not set.
Issue #1847 asserts that this is not a subversion issue.
I wish to dispute that.
1) Subversion claims to be able to handle binary data.
But this problem shows that it cannot handle binary data
in the filename, even when the host OS allows it.
2) If a project has filenames in more than one character set,
I don't think it is the revision control system's place
to disallow that. A revision control system should be able
to accept any file, and give it back in exactly the same way.
3) This error causes "svn status" to abort, not reporting the
status of the remaining files.
4) This error causes "svn add" to abort, leaving the directory
in some sort of half-baked state, which svn revert will not revert.
Even if 1 and 2 are disputed, surely there must be a more
graceful way to handle the error, such as skipping the file,
or rolling back the transaction.
~ John Williams
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Can't recode string
Posted by Cédric Girard <ce...@gmail.com>.
My solution for french accented chars : not use them!
But the real solution is to ugrade to the 1.10 (dump/clean no-utf8-char/import)
Cédric
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Can't recode string
Posted by Patrick Smears <pa...@ensoft.co.uk>.
On Wed, 13 Oct 2004, Scott Palmer wrote:
>
> On Oct 13, 2004, at 3:01 PM, Patrick Smears wrote:
>
> > OK, I confess I know next to nothing about Macs and less about OS X,
> > but
> > given that nobody else has replied, let's try some standard-ish Unix
> > stuff
> > to see if that helps...
> >
> > (0) Try the following commands and see what they display:
> >
> > locale -a
> > locale charmap
>
> $ locale -a
> -bash: locale: command not found
>
> Not looking good is it ;)
No ;)
... but the output of the other commands may yet shed some light...
Patrick
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Can't recode string
Posted by Scott Palmer <sc...@2connected.org>.
On Oct 13, 2004, at 3:01 PM, Patrick Smears wrote:
> OK, I confess I know next to nothing about Macs and less about OS X,
> but
> given that nobody else has replied, let's try some standard-ish Unix
> stuff
> to see if that helps...
>
> (0) Try the following commands and see what they display:
>
> locale -a
> locale charmap
$ locale -a
-bash: locale: command not found
Not looking good is it ;)
Scott
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Can't recode string
Posted by christian unger <cu...@apple.com>.
On 13.10.2004, at 21:39, Christian Unger wrote:
>> While the above may not solve the problem, they may provide some
>> clues as
>> to what might be needed...
>>
>> Patrick
>
> Thanks Patrick!
> I will report what I find out.
> with best regards
none of the locale environment variables are set under Mac OS X nor do
the utilities exist (locale...) but (as mentioned earlier):
http://developer.apple.com/documentation/MacOSX/Conceptual/
BPRuntimeConfig/Concepts/EnvironmentVars.html
> User Session Environment Variables
>
> Mac OS X supports the definition of environment variables in the scope
> of the current user session. On login, the loginwindow application
> looks for a special property list file with the name
> environment.plist. This file must be located in a directory called
> .MacOSX at the root of the user’s home directory. The path to this
> file is as follows:
>
> ~/.MacOSX/environment.plist
>
> If an environment.plist file exists, loginwindow looks for keys that
> are children of the root element. For each of these keys, loginwindow
> registers an environment variable of the same name and assigns it the
> value of the key. This file supports only the definition of
> environment variables. You cannot use this file to execute other forms
> of script code. The format of the file is the same XML format as other
> property list files, with each key in the file containing a string
> value.
setting this:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN"
"http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>LC_CTYPE</key>
<string>la_LN.UTF-8</string>
<key>cunger</key>
<string>1</string>
</dict>
</plist>
resolves the problem, I can check in and checkout files with japanese
filenames, without getting the "Can't recode string" error
I created this plist on both the client and the server. I am not using
the web interface but the svnserver.
svnX has problems displaying the japanese filenames though.
mit freundlichen Gruessen / with best regards
christian unger
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Can't recode string
Posted by Christian Unger <ch...@mac.com>.
On 13.10.2004, at 21:01, Patrick Smears wrote:
> While the above may not solve the problem, they may provide some clues
> as
> to what might be needed...
>
> Patrick
Thanks Patrick!
I will report what I find out.
with best regards
christian unger
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Can't recode string
Posted by Patrick Smears <pa...@ensoft.co.uk>.
On Tue, 12 Oct 2004, Christian Unger wrote:
>
> On 08.10.2004, at 08:42, christian unger wrote:
>
> > exactly my question:
> > Mac OS X comes with all locales preinstalled and is able to display
> > jp, fr, zh characters side by side even in the terminal window.
> > Which Environment variable does svn rely upon and how do I tweak it in
> > OS X ?
>
> yet another question:
> I found this on
> http://developer.apple.com/documentation/MacOSX/Conceptual/
> BPRuntimeConfig/Concepts/EnvironmentVars.html
>
>
> > User Session Environment Variables
> >
> > Mac OS X supports the definition of environment variables in the scope
> > of the current user session. On login, the loginwindow application
> > looks for a special property list file with the name
> > environment.plist. This file must be located in a directory called
> > .MacOSX at the root of the users home directory. The path to this
> > file is as follows:
> >
> > ~/.MacOSX/environment.plist
> >
> > If an environment.plist file exists, loginwindow looks for keys that
> > are children of the root element. For each of these keys, loginwindow
> > registers an environment variable of the same name and assigns it the
> > value of the key. This file supports only the definition of
> > environment variables. You cannot use this file to execute other forms
> > of script code. The format of the file is the same XML format as other
> > property list files, with each key in the file containing a string
> > value.
>
> Could this be the solution ? and if so, could someone tell me which
> variables I would have to set in this xml file to tell svn how to encode
> the japanese chars ? thanx in advance :-)
OK, I confess I know next to nothing about Macs and less about OS X, but
given that nobody else has replied, let's try some standard-ish Unix stuff
to see if that helps...
(0) Try the following commands and see what they display:
locale -a
locale charmap
(1) Try setting the variable "LANG" to one of the locales listed by
"locale -a" - for example, among the options on my Linux system
are: ja_JP, ja_JP.eucjp, ja_JP.ujis, ja_JP.utf8
(2) I assume the Japanese characters are displayed fine when you type
"ls"... if so, find out how the names are being encoded as follows:
ls <filename> | od -tx1
While the above may not solve the problem, they may provide some clues as
to what might be needed...
Patrick
--
The easy way to type accents in Windows: http://www.frkeys.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Can't recode string
Posted by Christian Unger <ch...@mac.com>.
On 08.10.2004, at 08:42, christian unger wrote:
> exactly my question:
> Mac OS X comes with all locales preinstalled and is able to display
> jp, fr, zh characters side by side even in the terminal window.
> Which Environment variable does svn rely upon and how do I tweak it in
> OS X ?
yet another question:
I found this on
http://developer.apple.com/documentation/MacOSX/Conceptual/
BPRuntimeConfig/Concepts/EnvironmentVars.html
> User Session Environment Variables
>
> Mac OS X supports the definition of environment variables in the scope
> of the current user session. On login, the loginwindow application
> looks for a special property list file with the name
> environment.plist. This file must be located in a directory called
> .MacOSX at the root of the user’s home directory. The path to this
> file is as follows:
>
> ~/.MacOSX/environment.plist
>
> If an environment.plist file exists, loginwindow looks for keys that
> are children of the root element. For each of these keys, loginwindow
> registers an environment variable of the same name and assigns it the
> value of the key. This file supports only the definition of
> environment variables. You cannot use this file to execute other forms
> of script code. The format of the file is the same XML format as other
> property list files, with each key in the file containing a string
> value.
Could this be the solution ? and if so, could someone tell me which
variables I would have to set in this xml file to tell svn how to
encode the japanese chars ?
thanx in advance :-)
with best regards
christian unger
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Can't recode string
Posted by christian unger <ch...@mac.com>.
On 07.10.2004, at 20:50, Ben Collins-Sussman wrote:
> It means that the svn client is unable to convert the UTF8 filename
> (coming from the repository) into your shell's native locale. Try
> setting your LANG environment variable to something that can deal with
> the filenames.
exactly my question:
Mac OS X comes with all locales preinstalled and is able to display jp,
fr, zh characters side by side even in the terminal window.
Which Environment variable does svn rely upon and how do I tweak it in
OS X ?
mit freundlichen Gruessen / with best regards
christian unger
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Can't recode string
Posted by Ben Collins-Sussman <su...@collab.net>.
On Oct 7, 2004, at 12:51 PM, Christian Unger wrote:
> Hi,
>
> I posted this a while ago. is there really no solution for the "Can't
> recode string" message ?
>
>
> [OS: Mac OS X 10.3.5]
> [Subversion: version 1.1(metissian)]
>
> French and Japanese filenames lead to commit abort with error "Can't
> recode string".
> which settings does svn need to be able to recode the strings of those
> files ?
>
It means that the svn client is unable to convert the UTF8 filename
(coming from the repository) into your shell's native locale. Try
setting your LANG environment variable to something that can deal with
the filenames.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org