You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apr.apache.org by Igor Galić <i....@brainsware.org> on 2011/07/23 04:24:12 UTC
APR trunk, xlate issues
Hey guys,
I've been debugging a make check failure of
Subversion trunk/apr trunk/serf trunk, together with danielsh.
This seems only to occur with FreeBSD:
igalic@beasdix ...ubversion/subversion/tests/libsvn_subr (svn)-[trunk:1149770] % ./subst_translate-test
subversion/tests/libsvn_subr/subst_translate-test.c:75: (apr_err=22)
subversion/libsvn_subr/subst.c:1890: (apr_err=22)
subversion/libsvn_subr/utf.c:817: (apr_err=22)
subversion/libsvn_subr/utf.c:779: (apr_err=22)
subversion/libsvn_subr/utf.c:624: (apr_err=22)
svn_tests: E000022: Non-ASCII character (code 199) detected, and unable to convert to/from UTF-8
FAIL: subst_translate-test 1: test svn_subst_translate_string2()
subversion/tests/libsvn_subr/subst_translate-test.c:124: (apr_err=22)
subversion/libsvn_subr/subst.c:1894: (apr_err=22)
subversion/libsvn_subr/utf.c:797: (apr_err=22)
subversion/libsvn_subr/utf.c:779: (apr_err=22)
subversion/libsvn_subr/utf.c:624: (apr_err=22)
svn_tests: E000022: Non-ASCII character (code 198) detected, and unable to convert to/from UTF-8
FAIL: subst_translate-test 2: test svn_subst_translate_string2(encoding = NULL)
PASS: subst_translate-test 3: test repairing svn_subst_translate_string2()
PASS: subst_translate-test 4: test svn_subst_translate_cstring2()
Note that the corresponding apr test succeeds on the same system:
igalic@beasdix /tmp/build/apr/test (svn)-[trunk:1149693] % ./testall testxlate
testxlate : SUCCESS
All tests passed.
So perhaps we aren't covering this thoroughfully enough?
i
--
Igor Galić
Tel: +43 (0) 664 886 22 883
Mail: i.galic@brainsware.org
URL: http://brainsware.org/
GPG: 571B 8B8A FC97 266D BDA3 EF6F 43AD 80A4 5779 3257
Re: APR trunk, xlate issues
Posted by "William A. Rowe Jr." <wr...@rowe-clan.net>.
On 7/22/2011 9:24 PM, Igor Galić wrote:
>
> So perhaps we aren't covering this thoroughfully enough?
I expect this is a quirk of bsd's implementation.
You might want to play with apr/test/internal/testucs.c as the
model for a rather exhaustive test of valid/invalid character
sequences for utf-8.
But I'd propose that the error caught says 'ASCII' meaning that
no values > 127 would be folded into or out of utf-8.
Re: APR trunk, xlate issues
Posted by Igor Galić <i....@brainsware.org>.
----- Original Message -----
> On Sat, Jul 23, 2011 at 01:36:07PM -0000, Igor Galić wrote:
> > > On OpenBSD I have to apply this patch to GNU iconv to avoid
> > > a similar problem in Subversion's prop_test 22 (with apr-util
> > > 1.3.12).
> > > But I don't recall the details.
> > >
> > > --- lib/aliases.gperf.orig Wed Oct 24 23:41:32 2007
> > > +++ lib/aliases.gperf Wed Oct 24 23:47:38 2007
> > > @@ -10,6 +10,7 @@ struct alias { int name; unsigned int
> > > encoding_index;
> > > %pic
> > > %%
> > > US-ASCII, ei_ascii
> > > +646, ei_ascii
> > > ASCII, ei_ascii
> > > ISO646-US, ei_ascii
> > > ISO_646.IRV:1991, ei_ascii
> >
> > I have *no* idea what that does.
>
> It makes GNU iconv recognise "646" as an alternative name for the
> "ASCII" encoding.
>
> > I applied it to the ports anyway,
> > recompiled/reinstalled the port and recompiled subversion to get
> > the exact same failure as before.
>
> Hmmm. I guess you'll have to dig down into the problem with a
> debugger.
> Can you find out which function is failing, and why?
> Basically, break at subversion/libsvn_subr/utf.c:check_non_ascii(),
> and look at the backtrace to figure out which call to
> get_ntou_xlate_handle_node() (in the same file) is failing.
> Then you'll have to drill down from there, maybe into iconv itself,
> and figure out why you're getting an error.
> This is nasty because there are many layers involved (svn uses apr
> uses iconv), but once you know where the error really comes from we
> should be able to fix this.
It seems to consistently fail when converting from ISO-8859-1
to UTF-8, same in utf-test, where it fails to translate from
"Edelwei\xdf" to "Edelwei\xc3\x9f"
Now I just have to find out why.
i
--
Igor Galić
Tel: +43 (0) 664 886 22 883
Mail: i.galic@brainsware.org
URL: http://brainsware.org/
GPG: 571B 8B8A FC97 266D BDA3 EF6F 43AD 80A4 5779 3257
Re: APR trunk, xlate issues
Posted by Igor Galić <i....@brainsware.org>.
----- Original Message -----
> On Sat, Jul 23, 2011 at 01:36:07PM -0000, Igor Galić wrote:
> > > On OpenBSD I have to apply this patch to GNU iconv to avoid
> > > a similar problem in Subversion's prop_test 22 (with apr-util
> > > 1.3.12).
> > > But I don't recall the details.
> > >
> > > --- lib/aliases.gperf.orig Wed Oct 24 23:41:32 2007
> > > +++ lib/aliases.gperf Wed Oct 24 23:47:38 2007
> > > @@ -10,6 +10,7 @@ struct alias { int name; unsigned int
> > > encoding_index;
> > > %pic
> > > %%
> > > US-ASCII, ei_ascii
> > > +646, ei_ascii
> > > ASCII, ei_ascii
> > > ISO646-US, ei_ascii
> > > ISO_646.IRV:1991, ei_ascii
> >
> > I have *no* idea what that does.
>
> It makes GNU iconv recognise "646" as an alternative name for the
> "ASCII" encoding.
>
> > I applied it to the ports anyway,
> > recompiled/reinstalled the port and recompiled subversion to get
> > the exact same failure as before.
>
> Hmmm. I guess you'll have to dig down into the problem with a
> debugger.
> Can you find out which function is failing, and why?
> Basically, break at subversion/libsvn_subr/utf.c:check_non_ascii(),
> and look at the backtrace to figure out which call to
> get_ntou_xlate_handle_node() (in the same file) is failing.
> Then you'll have to drill down from there, maybe into iconv itself,
> and figure out why you're getting an error.
> This is nasty because there are many layers involved (svn uses apr
> uses iconv), but once you know where the error really comes from we
> should be able to fix this.
It seems to consistently fail when converting from ISO-8859-1
to UTF-8, same in utf-test, where it fails to translate from
"Edelwei\xdf" to "Edelwei\xc3\x9f"
Now I just have to find out why.
i
--
Igor Galić
Tel: +43 (0) 664 886 22 883
Mail: i.galic@brainsware.org
URL: http://brainsware.org/
GPG: 571B 8B8A FC97 266D BDA3 EF6F 43AD 80A4 5779 3257
Re: APR trunk, xlate issues
Posted by Stefan Sperling <st...@elego.de>.
On Sat, Jul 23, 2011 at 01:36:07PM -0000, Igor Galić wrote:
> > On OpenBSD I have to apply this patch to GNU iconv to avoid
> > a similar problem in Subversion's prop_test 22 (with apr-util
> > 1.3.12).
> > But I don't recall the details.
> >
> > --- lib/aliases.gperf.orig Wed Oct 24 23:41:32 2007
> > +++ lib/aliases.gperf Wed Oct 24 23:47:38 2007
> > @@ -10,6 +10,7 @@ struct alias { int name; unsigned int
> > encoding_index;
> > %pic
> > %%
> > US-ASCII, ei_ascii
> > +646, ei_ascii
> > ASCII, ei_ascii
> > ISO646-US, ei_ascii
> > ISO_646.IRV:1991, ei_ascii
>
> I have *no* idea what that does.
It makes GNU iconv recognise "646" as an alternative name for the
"ASCII" encoding.
> I applied it to the ports anyway,
> recompiled/reinstalled the port and recompiled subversion to get
> the exact same failure as before.
Hmmm. I guess you'll have to dig down into the problem with a debugger.
Can you find out which function is failing, and why?
Basically, break at subversion/libsvn_subr/utf.c:check_non_ascii(),
and look at the backtrace to figure out which call to
get_ntou_xlate_handle_node() (in the same file) is failing.
Then you'll have to drill down from there, maybe into iconv itself,
and figure out why you're getting an error.
This is nasty because there are many layers involved (svn uses apr
uses iconv), but once you know where the error really comes from we
should be able to fix this.
Re: APR trunk, xlate issues
Posted by Igor Galić <i....@brainsware.org>.
> Which iconv implementation is APR using?
> This could be some problem in iconv.
GNU libiconv 1.13.1
> On OpenBSD I have to apply this patch to GNU iconv to avoid
> a similar problem in Subversion's prop_test 22 (with apr-util
> 1.3.12).
> But I don't recall the details.
>
> --- lib/aliases.gperf.orig Wed Oct 24 23:41:32 2007
> +++ lib/aliases.gperf Wed Oct 24 23:47:38 2007
> @@ -10,6 +10,7 @@ struct alias { int name; unsigned int
> encoding_index;
> %pic
> %%
> US-ASCII, ei_ascii
> +646, ei_ascii
> ASCII, ei_ascii
> ISO646-US, ei_ascii
> ISO_646.IRV:1991, ei_ascii
I have *no* idea what that does. I applied it to the ports anyway,
recompiled/reinstalled the port and recompiled subversion to get
the exact same failure as before.
--
Igor Galić
Tel: +43 (0) 664 886 22 883
Mail: i.galic@brainsware.org
URL: http://brainsware.org/
GPG: 571B 8B8A FC97 266D BDA3 EF6F 43AD 80A4 5779 3257
Re: APR trunk, xlate issues
Posted by Igor Galić <i....@brainsware.org>.
> Which iconv implementation is APR using?
> This could be some problem in iconv.
GNU libiconv 1.13.1
> On OpenBSD I have to apply this patch to GNU iconv to avoid
> a similar problem in Subversion's prop_test 22 (with apr-util
> 1.3.12).
> But I don't recall the details.
>
> --- lib/aliases.gperf.orig Wed Oct 24 23:41:32 2007
> +++ lib/aliases.gperf Wed Oct 24 23:47:38 2007
> @@ -10,6 +10,7 @@ struct alias { int name; unsigned int
> encoding_index;
> %pic
> %%
> US-ASCII, ei_ascii
> +646, ei_ascii
> ASCII, ei_ascii
> ISO646-US, ei_ascii
> ISO_646.IRV:1991, ei_ascii
I have *no* idea what that does. I applied it to the ports anyway,
recompiled/reinstalled the port and recompiled subversion to get
the exact same failure as before.
--
Igor Galić
Tel: +43 (0) 664 886 22 883
Mail: i.galic@brainsware.org
URL: http://brainsware.org/
GPG: 571B 8B8A FC97 266D BDA3 EF6F 43AD 80A4 5779 3257
Re: APR trunk, xlate issues
Posted by Stefan Sperling <st...@elego.de>.
On Sat, Jul 23, 2011 at 02:24:12AM -0000, Igor Galić wrote:
> Hey guys,
>
> I've been debugging a make check failure of
> Subversion trunk/apr trunk/serf trunk, together with danielsh.
>
> This seems only to occur with FreeBSD:
>
> igalic@beasdix ...ubversion/subversion/tests/libsvn_subr (svn)-[trunk:1149770] % ./subst_translate-test
> subversion/tests/libsvn_subr/subst_translate-test.c:75: (apr_err=22)
> subversion/libsvn_subr/subst.c:1890: (apr_err=22)
> subversion/libsvn_subr/utf.c:817: (apr_err=22)
> subversion/libsvn_subr/utf.c:779: (apr_err=22)
> subversion/libsvn_subr/utf.c:624: (apr_err=22)
> svn_tests: E000022: Non-ASCII character (code 199) detected, and unable to convert to/from UTF-8
> FAIL: subst_translate-test 1: test svn_subst_translate_string2()
> subversion/tests/libsvn_subr/subst_translate-test.c:124: (apr_err=22)
> subversion/libsvn_subr/subst.c:1894: (apr_err=22)
> subversion/libsvn_subr/utf.c:797: (apr_err=22)
> subversion/libsvn_subr/utf.c:779: (apr_err=22)
> subversion/libsvn_subr/utf.c:624: (apr_err=22)
> svn_tests: E000022: Non-ASCII character (code 198) detected, and unable to convert to/from UTF-8
> FAIL: subst_translate-test 2: test svn_subst_translate_string2(encoding = NULL)
> PASS: subst_translate-test 3: test repairing svn_subst_translate_string2()
> PASS: subst_translate-test 4: test svn_subst_translate_cstring2()
>
> Note that the corresponding apr test succeeds on the same system:
>
> igalic@beasdix /tmp/build/apr/test (svn)-[trunk:1149693] % ./testall testxlate
> testxlate : SUCCESS
> All tests passed.
>
> So perhaps we aren't covering this thoroughfully enough?
Which iconv implementation is APR using?
This could be some problem in iconv.
On OpenBSD I have to apply this patch to GNU iconv to avoid
a similar problem in Subversion's prop_test 22 (with apr-util 1.3.12).
But I don't recall the details.
--- lib/aliases.gperf.orig Wed Oct 24 23:41:32 2007
+++ lib/aliases.gperf Wed Oct 24 23:47:38 2007
@@ -10,6 +10,7 @@ struct alias { int name; unsigned int encoding_index;
%pic
%%
US-ASCII, ei_ascii
+646, ei_ascii
ASCII, ei_ascii
ISO646-US, ei_ascii
ISO_646.IRV:1991, ei_ascii