You are viewing a plain text version of this content. The canonical link for it is here.
Posted to c-users@xalan.apache.org by Ashley Zinyk <az...@ca.ibm.com> on 2008/02/01 11:30:29 UTC
Re: MsgCreator produces weird arguments to genccode
David Bertoni <db...@apache.org> wrote on 12/22/2007 11:18:24 AM:
> Ashley Zinyk wrote:
> > /home/torxslt/xsltbts_client/working/20071219205227/2187/xml4c5_7_0-
> > suse_91_zOS_64-gcc_333/icu/sbin/genccode
> > -n xalanMsg_.. -d ../../../nls/icu ../../../nls/icu/xalanMsg_en_US.res
-f
> > .._.._.._nls_icu_xalanMsg_en_US_res
> >
> > generating C code for ../../../nls/icu/xalanMsg_en_US.res
> >
> > I think that the arguments passed to genccode and gcc, like
> > .._.._.._nls_icu_xalanMsg_en_US_res, are wrong, but I'm not sure what's
> > causing the problem.
>
> I think that's a result of the call to pkgdata, since it calls gencode.
Yes, I narrowed it down to that call (to pkgdata). I can't figure out,
however, how pkgdata calls genccode, or where it generates the arguments.
Googling "pkgdata" and "genccode" doesn't help much. In previous builds,
the arguments looked like this:
"xml4c5_6_3-suse_91_zOS_64-gcc_333/icu/sbin/genccode -d ../../../nls/icu
../../../nls/icu/xalanMsg_en_US.res". There weren't any -n or -f
arguments. If I knew where those arguments were being generated, I would
have a place to start.
> If you're using any version of ICU later than 3.2.1, the Makefiles for
> using ICU message bundles are broken. What version are you using?
I'm trying to use ICU version 3.8. In what way are the Makefiles broken,
and which ones? What can I do to update them to use ICU 3.8?
Ashley Zinyk
XSLT Development
Re: MsgCreator incompatible with ICU 3.8
Posted by David Bertoni <db...@apache.org>.
Ashley Zinyk wrote:
> Hi,
>
> David Bertoni <db...@apache.org> wrote on 02/01/2008 12:37:36 PM:
>> In newer versions of the ICU, the _entire_ path becomes the name of the
>> resource. You can see that if you take a look at the messages shared
>> library using a hex editor.
>>
>> I think we need to modify the build process so we no longer use the full
>> relative path in XalanMessages_1_10_en_US.lst, although I'm not sure of
>> that. At any rate, it needs some investigation, but I just don't have
> time
>> to do it right now.
>>
>> If you really need to get this fixed, I suggest you read the ICU user
> guide
>> to see if there's more information about naming resources.
>
> I took your advice and I have it working on all my platforms now, although
> there were some stumbling blocks along the way. The first thing I tried
> was to just generate all the resources in my local directory (
> c/src/xalanc/Utils instead of c/nls/icu), but when I did that, it
> complained that xalanMsg_en_US.res depends on the missing resource
> xalanMsg_en.res.
Yes, the dependencies probably use a full path, rather than just assuming
the current directory.
>
> If you look at this documentation (
> http://icu-project.org/userguide/ResourceManagement.html), it says that ICU
> tries to find the most specific available locale, like en_US, and then
> proceeds to try more general locales if it fails. In this case, if it
> can't find American English, it would try general English (en) and
> eventually the "root" locale. Given that, it seems completely sensible to
> me that the build system would complain that we don't provide it with the
> general English or root .res files. I wasn't able to figure out why it
> complained when I tried to build in c/src/xalanc/Utils (which I think is
> proper behavior), but mysteriously succeeded when I built in c/nls/icu. If
> anyone can understand why this might be path-sensitive, please let me know.
I'm not sure I understand why we have to provide a root locale since we
supply the locale explicitly when we load the messages. It's not a problem
if we _do_ provide the root locale, but we don't want it to be the reason
why things are working now.
>
> To fix these problems, I altered MsgCreator.cpp so that, in addition to
> accepting locales of the form <language>_<country> and <language>, it now
> also accepts the locale "root". Then, I changed
> c/src/xalanc/Utils/Makefile.in so that it would build root.res from
> XalanMsg_en_US.xlf (I thought US English would be an adequate default,
> considering it's the only message catalog I had, and considering that we
> didn't have any default before).
As I said, this shouldn't be necessary. I think there's still something
wrong with the way we're building the .res file, because we shouldn't have
to fall back to US English messages, even though it's fine if they are the
root once we get things working again.
>
> I also removed the leading "xalanMsg_", since
> http://source.icu-project.org/repos/icu/icu/tags/release-3-6/readme.html#News
> seems to forbid it. Again, I'm not sure how my builds managed to succeed
> without that change.
OK, so that explains it. I believe the ICU is interpreting the relative
path we put in res-file-list.txt as the path to the resources within the
ICU messages bundle. I think we need to figure out a way to put just a
file name in that file, and not a path. That might be as simple as
tweaking src/xalanc/Utils/Makefile.in:
$ svn diff Makefile.in
Index: Makefile.in
===================================================================
--- Makefile.in (revision 619123)
+++ Makefile.in (working copy)
@@ -100,7 +100,7 @@
$(MESSAGELIB) : $(XSL_NLS_DIR)/icu/$(LOCALE).txt
$(GENRB) -p $(LOC_PROJECT_NAME) -d $(XSL_NLS_DIR)/icu -i
$(XSL_NLS_DIR)/icu $(XSL_NLS_DIR)/icu/$(LOCALE).txt
- echo $(XSL_NLS_DIR)/icu/$(LOC_PROJECT_NAME)_$(LOCALE).res >
$(XSL_NLS_DIR)/icu/res-file-list.txt
+ echo $(LOC_PROJECT_NAME)_$(LOCALE).res >
$(XSL_NLS_DIR)/icu/res-file-list.txt
$(PKGDATA) --name $(LOC_PROJECT_NAME) -T $(XSL_NLS_DIR)/icu -v -O
$(ICUROOT)/lib/icu/Makefile.inc --mode dll -d $(XSL_LIB_DIR)
$(XSL_NLS_DIR)/icu/res-file-list.txt -M "prefix=$(ICUROOT) TARGET=$(LOC_LIB)"
$(LN) -fs $(LOC_LIB) $(XSL_LIB_DIR)/$(LOC_LIBNAME)$(SHLIBSUFFIX)
ifeq ($(PLATFORM), AIX)
Dave
MsgCreator incompatible with ICU 3.8
Posted by Ashley Zinyk <az...@ca.ibm.com>.
Hi,
David Bertoni <db...@apache.org> wrote on 02/01/2008 12:37:36 PM:
> In newer versions of the ICU, the _entire_ path becomes the name of the
> resource. You can see that if you take a look at the messages shared
> library using a hex editor.
>
> I think we need to modify the build process so we no longer use the full
> relative path in XalanMessages_1_10_en_US.lst, although I'm not sure of
> that. At any rate, it needs some investigation, but I just don't have
time
> to do it right now.
>
> If you really need to get this fixed, I suggest you read the ICU user
guide
> to see if there's more information about naming resources.
I took your advice and I have it working on all my platforms now, although
there were some stumbling blocks along the way. The first thing I tried
was to just generate all the resources in my local directory (
c/src/xalanc/Utils instead of c/nls/icu), but when I did that, it
complained that xalanMsg_en_US.res depends on the missing resource
xalanMsg_en.res.
If you look at this documentation (
http://icu-project.org/userguide/ResourceManagement.html), it says that ICU
tries to find the most specific available locale, like en_US, and then
proceeds to try more general locales if it fails. In this case, if it
can't find American English, it would try general English (en) and
eventually the "root" locale. Given that, it seems completely sensible to
me that the build system would complain that we don't provide it with the
general English or root .res files. I wasn't able to figure out why it
complained when I tried to build in c/src/xalanc/Utils (which I think is
proper behavior), but mysteriously succeeded when I built in c/nls/icu. If
anyone can understand why this might be path-sensitive, please let me know.
To fix these problems, I altered MsgCreator.cpp so that, in addition to
accepting locales of the form <language>_<country> and <language>, it now
also accepts the locale "root". Then, I changed
c/src/xalanc/Utils/Makefile.in so that it would build root.res from
XalanMsg_en_US.xlf (I thought US English would be an adequate default,
considering it's the only message catalog I had, and considering that we
didn't have any default before).
I also removed the leading "xalanMsg_", since
http://source.icu-project.org/repos/icu/icu/tags/release-3-6/readme.html#News
seems to forbid it. Again, I'm not sure how my builds managed to succeed
without that change.
If these changes sound reasonable, I can put a patch with my changes on
JIRA for review.
Ashley Zinyk
XSLT Development
Re: MsgCreator produces weird arguments to genccode
Posted by David Bertoni <db...@apache.org>.
Ashley Zinyk wrote:
> David Bertoni <db...@apache.org> wrote on 12/22/2007 11:18:24 AM:
>> Ashley Zinyk wrote:
>>> /home/torxslt/xsltbts_client/working/20071219205227/2187/xml4c5_7_0-
>>> suse_91_zOS_64-gcc_333/icu/sbin/genccode
>>> -n xalanMsg_.. -d ../../../nls/icu ../../../nls/icu/xalanMsg_en_US.res
> -f
>>> .._.._.._nls_icu_xalanMsg_en_US_res
>>>
>>> generating C code for ../../../nls/icu/xalanMsg_en_US.res
>>>
>>> I think that the arguments passed to genccode and gcc, like
>>> .._.._.._nls_icu_xalanMsg_en_US_res, are wrong, but I'm not sure what's
>>> causing the problem.
>> I think that's a result of the call to pkgdata, since it calls gencode.
>
> Yes, I narrowed it down to that call (to pkgdata). I can't figure out,
> however, how pkgdata calls genccode, or where it generates the arguments.
> Googling "pkgdata" and "genccode" doesn't help much. In previous builds,
> the arguments looked like this:
> "xml4c5_6_3-suse_91_zOS_64-gcc_333/icu/sbin/genccode -d ../../../nls/icu
> ../../../nls/icu/xalanMsg_en_US.res". There weren't any -n or -f
> arguments. If I knew where those arguments were being generated, I would
> have a place to start.
I assume it calls it by spawning a process using the system() call, but I
don't know for sure. Why don't you take a look at the source code for the
ICU if you're curious.
>
>> If you're using any version of ICU later than 3.2.1, the Makefiles for
>> using ICU message bundles are broken. What version are you using?
>
> I'm trying to use ICU version 3.8. In what way are the Makefiles broken,
> and which ones? What can I do to update them to use ICU 3.8?
We were relying (unknowingly) on an option that was deprecated in ICU 3.2
and has since disappeared. If you take a look at a build on Windows using
3.2.1, you'll see the following:
2>------ Build started: Project: XalanMsgLibICU, Configuration: Debug Win32
------
2>Processing XLIFF file
2>Processing ICU resource file
2>Processing file "..\..\..\..\..\Build\Win32\VC8\Debug\Nls\Include\en_US.txt"
2>autodetected encoding UTF-16LE
2> parsing table (null) at line 20
2> resource xalanc at line 21
2> array xalanc at line 21
2>Creating resource list
2>Building resource DLL
2>Microsoft (R) Program Maintenance Utility Version 8.00.50727.762
2>Copyright (C) Microsoft Corporation. All rights reserved.
2> "V:\icu\bin\gencmn.exe" -d "..\..\..\..\..\Build\Win32\VC8\Debug" -E -n
"XalanMessages_1_10" 0 C:\DOCUME~1\dbertoni\LOCALS~1\Temp\nm3C4.tmp
2>generating object code for
..\..\..\..\..\Build\Win32\VC8\Debug\XalanMessages_1_10.dat
2> link.exe /nologo
/out:"..\..\..\..\..\Build\Win32\VC8\Debug\XalanMessages_1_10.DLL" /DLL
/NOENTRY /base:"0x4ad00000"
/implib:"..\..\..\..\..\Build\Win32\VC8\Debug\XalanMessages_1_10.lib"
"..\..\..\..\..\Build\Win32\VC8\Debug\Utils\XalanMsgLibICU\XalanMessages_1_10_dat.obj"
2> Creating library
..\..\..\..\..\Build\Win32\VC8\Debug\XalanMessages_1_10.lib and object
..\..\..\..\..\Build\Win32\VC8\Debug\XalanMessages_1_10.exp
2># ICUROOT is V:\icu
2># Reading
..\..\..\..\..\Build\Win32\VC8\Debug\Utils\XalanMsgLibICU\XalanMessages_1_10_en_US.lst..
2># Output .DLL file:
..\..\..\..\..\Build\Win32\VC8\Debug\XalanMessages_1_10.DLL
2># Output file: ..\..\..\..\..\Build\Win32\VC8\Debug\XalanMessages_1_10.dat
2>nmake -f
"..\..\..\..\..\Build\Win32\VC8\Debug\Utils\XalanMsgLibICU\XalanMessages_1_10_dll.mak"
2>Warning: You are using the deprecated -O option
2> You can fix this warning by installing pkgdata, gencmn and genccode
2> into the same directory and not specifying the -O option to pkgdata.
2>Warning: Found path 'XalanMessages_1_10_' in file name. Assuming
compatibility (-E) mode.
It's this last warning that indicates the problem. if you look at the
intermediate file XalanMessages_1_10_en_US.lst, you'll see this:
..\..\..\..\..\Build\Win32\VC8\Debug\Utils\XalanMsgLibICU\XalanMessages_1_10_en_US.res
That's what's causing the warning, and turning out "compatibility mode."
In newer versions of the ICU, the _entire_ path becomes the name of the
resource. You can see that if you take a look at the messages shared
library using a hex editor.
I think we need to modify the build process so we no longer use the full
relative path in XalanMessages_1_10_en_US.lst, although I'm not sure of
that. At any rate, it needs some investigation, but I just don't have time
to do it right now.
If you really need to get this fixed, I suggest you read the ICU user guide
to see if there's more information about naming resources.
Dave