You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by David Kramer <da...@thekramers.net> on 2005/09/07 15:29:45 UTC
Re: Locale problem: Can't convert string from native encoding to
'UTF-8'
On Tue, September 6, 2005 3:30 pm, David Kramer wrote:
> Background:
> - I'm running svnserve 1.1.0 on my old server using berkeley back end for the
> repository. (SuSE 9.0)
>
> - I just built a new server, now running svnserve 1.2.1, using fsfs for the
> repository (SUSE 9.3, with subversion server and libraries updated from
> ftp.suse.com/pub/projects)
>
> - I dumped the repositories from the old server and loaded them on the new
> server.
>
> - I twiddled my router so now all svn:// requests went to the new server.
>
> After doing that, I was able to check out and in with the new repository (this
> is as of a day or two ago). Now, I get the following error:
>
> david@deepthink:/devel/agilerules/web/private/htdocs/worknotes> svn commit -m
> "web private: removed worknotes/htmlunit"
> Deleting worknotes/htmlunit
> svn: Commit failed (details follow):
> svn: Can't convert string from native encoding to 'UTF-8':
> @?\217?d?\14?\184?\174
>
> - All files and filenames in version control are plain text or OpenOffice
> documents. All plain text files are normal ASCII US characters.
>
> Now, I did some RTFM and some STFW, and noticed that (1) part of the 1.2
> series changes was added localization, and (2) the way you set the
> localization of the server is by exporting LANG to the correct setting in the
> script that starts up svnserve, and I've now done that, setting/exporting both
> LANG and LC_ALL to en_US.UTF-8 in /etc/init.d/svnserve. This did not seem to
> help.
>
> We tried upgrading the client to the latest, and that didn't seem to help
> either.
>
> After changing the locale on svnserve, do I need to do the load operations
> again?
>
> Do I need to make new working copies?
>
> One bit of funkyness is that I had libapr0-2.0.53-9, to match
> apache2-2.0.53-9.2. The subversion-server-1.2.1-1.1. said to install
> libapr0-2.0.54-7.1.i586.rpm, but when I did so, apache would segfault on all
> client requests, so I reverted to libapr0-2.0.53-9. Both apps seemed to work
> after that, but I'm not sure if that's involved or not. Maybe whether using
> that version of svnserve with that version of of libapr is a separate
> question.
>
>
> Thanks. We can't check in anything until this is resolved, so please get back
> to me with any tips as soon as possible. Thanks.
We've tried a few more things, with no luck.
- I tried checking out a fresh working copy, but I got the same "Can't convert
string from native encoding to 'UTF-8'"
- I tried wiping out the repository and reloading it, now that svnserve is
running UTF-8, but reloading failed, with
"uninew:/data/subversion/agilerules # svnadmin load web <
/tmp/agilerules.web.dump
<<< Started new transaction, based on original revision 1
* adding path : devel ... done.
* adding path : live ... done.
svnadmin: Valid UTF-8 data
(hex:)
followed by invalid UTF-8 sequence
(hex: a0 d1 07 08)
"
Is there a way to convert the dump to UTF-8 (it's all US ASCII, AFAIK)?
Is there a way to load the dump as native, and get the svnserve client to work
with it?
Thanks again. We at a dead stop over this.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Locale problem: Can't convert string from native encoding to 'UTF-8'
Posted by Joshua Varner <jl...@gmail.com>.
On 9/7/05, David Kramer <da...@thekramers.net> wrote:
> On Tue, September 6, 2005 3:30 pm, David Kramer wrote:
> > Background:
> > - I'm running svnserve 1.1.0 on my old server using berkeley back end for the
> > repository. (SuSE 9.0)
> >
> > - I just built a new server, now running svnserve 1.2.1, using fsfs for the
> > repository (SUSE 9.3, with subversion server and libraries updated from
> > ftp.suse.com/pub/projects)
> >
> > - I dumped the repositories from the old server and loaded them on the new
> > server.
> >
> > - I twiddled my router so now all svn:// requests went to the new server.
> >
> > After doing that, I was able to check out and in with the new repository (this
> > is as of a day or two ago). Now, I get the following error:
> >
> > david@deepthink:/devel/agilerules/web/private/htdocs/worknotes> svn commit -m
> > "web private: removed worknotes/htmlunit"
> > Deleting worknotes/htmlunit
> > svn: Commit failed (details follow):
> > svn: Can't convert string from native encoding to 'UTF-8':
> > @?\217?d?\14?\184?\174
> >
> > - All files and filenames in version control are plain text or OpenOffice
> > documents. All plain text files are normal ASCII US characters.
> >
The only things that should be converted to UTF-8 are filenames and
some properties. Since you already rolled your own server, you might
try building a client with --enable-maintainer-mode to get the line number
of the error.
> >
> > We tried upgrading the client to the latest, and that didn't seem to help
> > either.
> >
> > After changing the locale on svnserve, do I need to do the load operations
> > again?
> >
> > Do I need to make new working copies?
> >
> > One bit of funkyness is that I had libapr0-2.0.53-9, to match
> > apache2-2.0.53-9.2. The subversion-server-1.2.1-1.1. said to install
> > libapr0-2.0.54-7.1.i586.rpm, but when I did so, apache would segfault on all
> > client requests, so I reverted to libapr0-2.0.53-9. Both apps seemed to work
> > after that, but I'm not sure if that's involved or not. Maybe whether using
> > that version of svnserve with that version of of libapr is a separate
> > question.
> >
> >
> > Thanks. We can't check in anything until this is resolved, so please get back
> > to me with any tips as soon as possible. Thanks.
>
> We've tried a few more things, with no luck.
>
> - I tried checking out a fresh working copy, but I got the same "Can't convert
> string from native encoding to 'UTF-8'"
>
> - I tried wiping out the repository and reloading it, now that svnserve is
> running UTF-8, but reloading failed, with
> "uninew:/data/subversion/agilerules # svnadmin load web <
> /tmp/agilerules.web.dump
> <<< Started new transaction, based on original revision 1
> * adding path : devel ... done.
> * adding path : live ... done.
> svnadmin: Valid UTF-8 data
> (hex:)
> followed by invalid UTF-8 sequence
> (hex: a0 d1 07 08)
> "
>
> Is there a way to convert the dump to UTF-8 (it's all US ASCII, AFAIK)?
>
> Is there a way to load the dump as native, and get the svnserve client to work
> with it?
>
> Thanks again. We at a dead stop over this.
>
We've had some problems with weird characters showing up in text files
in conversions (non-subversion related), You could try the script below.
It'll be safe on any plain text data.
You can also try setting LANG=C on the server. This is a fix for some
UTF-8 conversion problems with perl, so maybe it would help.
Just some suggestions, haven't tried them for this problem, but it's
better than nothing.
Josh
#!/bin/sh
#
# perform some additional needed .doc to .txt translations
#
o342=`echo -ne '\342'`
o200=`echo -ne '\200'`
o231=`echo -ne '\231'`
o241=`echo -ne '\241'`
o226=`echo -ne '\226'`
o234=`echo -ne '\234'`
o235=`echo -ne '\235'`
o302=`echo -ne '\302'`
o255=`echo -ne '\255'`
sed -e "s/$o342$o200$o231/'/g" \
-e "s/$o342$o226$o241//g" \
-e "s/$o342$o200$o234/\"/g" \
-e "s/$o342$o200$o235/\"/g" \
-e "s/$o302$o255/\-/g" \
"$1" |
tr '\200-\377\014' ' '
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org
Re: Locale problem: Can't convert string from native encoding to 'UTF-8'
Posted by Ryan Schmidt <su...@ryandesign.com>.
On Sep 7, 2005, at 17:29, David Kramer wrote:
> Is there a way to convert the dump to UTF-8 (it's all US ASCII,
> AFAIK)?
Well, ASCII is a subset of UTF-8. If you have a collection of ASCII
characters, then you also by definition have a collection of UTF-8
characters. There is no conversion to be done.
So the fact that you have an error signifies that you do not have
exclusively ASCII characters.
> svnadmin: Valid UTF-8 data
> (hex:)
> followed by invalid UTF-8 sequence
> (hex: a0 d1 07 08)
Let's see here.... In the ISO-8859 character sets (all of them), A0
is a non-breaking space. D1 is (in ISO-8859-1, -3 and -9) is a
capital N with tilde ("Ñ") (and in other ISO-8859 sets, it's
undefined). 07 is a bell, and 08 is a backspace. I don't know
svnadmin well enough to know what kind of data it's talking about
here. If that data is the contents of a binary file, then that
sequence of bytes is conceivable, though it certainly won't conform
to UTF-8, so not sure why svnadmin would think it should. If that's
part of a text file's contents, or part of a filename or some
properties, then that's a strange sequence of characters indeed. It
almost sounds like something is corrupted somewhere. Perhaps you can
open the dump file in a good editor (as in one that doesn't have to
load the entire file into memory all at once) and search for the Ñ
and see where it is. If you have binary files in your repository,
then this may be more complicated (will probably give you many false-
positives).
> svn: Can't convert string from native encoding to 'UTF-8':
> @?\217?d?\14?\184?\174
217 is D9 which is a capital U with grave accent ("Ù"). 14 is
apparently the shift-out code, which I had never heard of until now.
184 is B8 which is a cedilla ("¸", the hook that usually appears
below a c). 174 is AE which is the registered trademark symbol ("®").
Same comment and advice as above.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org