You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stdcxx.apache.org by "Andrew Black (JIRA)" <ji...@apache.org> on 2006/08/18 18:02:13 UTC
[jira] Created: (STDCXX-277) Locale stage 2 test failure
Locale stage 2 test failure
---------------------------
Key: STDCXX-277
URL: http://issues.apache.org/jira/browse/STDCXX-277
Project: C++ Standard Library
Issue Type: Bug
Components: Utilities
Environment: Multiple, including Linux 2.6.14.5 + gcc 3.2.3
Reporter: Andrew Black
Attachments: en_US.ISO-8859-1.sh.out, out.1
To test the locales, the run_locale_utils.sh script is used, residing in the etc/config subdirectory of the source directory.
During this process, the source file in the etc/nls subdirectory is first transformed into a locale database using the localedef utility, then is converted back to a source file using the locale utility. This process is repeated twice more, with each following stage relying on the source file from the previous itteration. The test then concludes by checking that the source files produced in the second and third stage are identical.
On some platforms, including my local box, this process fails for most locales when the localedef utility in the second stage is unable to process the output of the source file generated by the locale utility in the first stage of the process. Output from running the en_US.ISO-8859-1.sh (wrapper) script with the '-d' flag on my box is attached as en_US.ISO-8859-1.sh.out. The failure message from the localedef utility in this case is as follows:
Error 399: decimal value in the range [0, 255) expected: \d383
When I search for the string '\d383' in the source file being parsed (attached as out.1), I find it in the LC_COLLATE section. It appears to me that either the output from the locale utility doesn't use the correct range, or that the valid range for the parser in localedef is incorrect.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (STDCXX-277) Locale stage 2 test failure
Posted by "Martin Sebor (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/STDCXX-277?page=comments#action_12431106 ]
Martin Sebor commented on STDCXX-277:
-------------------------------------
I think the source of the problem is that the locale utility prints out the collating weights as characters in the decimal notation regardless of their value and localedef rejects them based on the POSIX requirement that each character in decimal notation represent a byte value. Quoting from http://www.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap07.html#tag_07_03:
A character can be represented as a decimal constant. A decimal constant shall be specified as the escape character
followed by a 'd' followed by two or three decimal digits. Each constant represents a byte value. Multi-byte values can be
represented by concatenated constants specified in byte order with the last constant specifying the least significant byte
of the character.
So weights greater than UCHAR_MAX should probably be formatted as multibyte characters instead. For example, 383 should be formatted as "\d01;\d127" and not as "\d383".
> Locale stage 2 test failure
> ---------------------------
>
> Key: STDCXX-277
> URL: http://issues.apache.org/jira/browse/STDCXX-277
> Project: C++ Standard Library
> Issue Type: Bug
> Components: Utilities
> Environment: Multiple, including Linux 2.6.14.5 + gcc 3.2.3
> Reporter: Andrew Black
> Attachments: en_US.ISO-8859-1.sh.out, out.1
>
>
> To test the locales, the run_locale_utils.sh script is used, residing in the etc/config subdirectory of the source directory.
> During this process, the source file in the etc/nls subdirectory is first transformed into a locale database using the localedef utility, then is converted back to a source file using the locale utility. This process is repeated twice more, with each following stage relying on the source file from the previous itteration. The test then concludes by checking that the source files produced in the second and third stage are identical.
> On some platforms, including my local box, this process fails for most locales when the localedef utility in the second stage is unable to process the output of the source file generated by the locale utility in the first stage of the process. Output from running the en_US.ISO-8859-1.sh (wrapper) script with the '-d' flag on my box is attached as en_US.ISO-8859-1.sh.out. The failure message from the localedef utility in this case is as follows:
> Error 399: decimal value in the range [0, 255) expected: \d383
> When I search for the string '\d383' in the source file being parsed (attached as out.1), I find it in the LC_COLLATE section. It appears to me that either the output from the locale utility doesn't use the correct range, or that the valid range for the parser in localedef is incorrect.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (STDCXX-277) [locale] invalid LC_COLLATE output
for weights > 255
Posted by "Martin Sebor (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/STDCXX-277?page=all ]
Martin Sebor resolved STDCXX-277.
---------------------------------
Fix Version/s: 4.2
Resolution: Fixed
Fixed with the referenced changes. The en_US.ISO-8859-1.sh script now completes with 0 assertions (tested on HP-UX 11.23 and Solaris 9).
> [locale] invalid LC_COLLATE output for weights > 255
> ----------------------------------------------------
>
> Key: STDCXX-277
> URL: http://issues.apache.org/jira/browse/STDCXX-277
> Project: C++ Standard Library
> Issue Type: Bug
> Components: Utilities
> Affects Versions: 4.1.2, 4.1.3
> Environment: Multiple, including Linux 2.6.14.5 + gcc 3.2.3
> Reporter: Andrew Black
> Assigned To: Martin Sebor
> Fix For: 4.2
>
> Attachments: en_US.ISO-8859-1.sh.out, out.1
>
>
> To test the locales, the run_locale_utils.sh script is used, residing in the etc/config subdirectory of the source directory.
> During this process, the source file in the etc/nls subdirectory is first transformed into a locale database using the localedef utility, then is converted back to a source file using the locale utility. This process is repeated twice more, with each following stage relying on the source file from the previous itteration. The test then concludes by checking that the source files produced in the second and third stage are identical.
> On some platforms, including my local box, this process fails for most locales when the localedef utility in the second stage is unable to process the output of the source file generated by the locale utility in the first stage of the process. Output from running the en_US.ISO-8859-1.sh (wrapper) script with the '-d' flag on my box is attached as en_US.ISO-8859-1.sh.out. The failure message from the localedef utility in this case is as follows:
> Error 399: decimal value in the range [0, 255) expected: \d383
> When I search for the string '\d383' in the source file being parsed (attached as out.1), I find it in the LC_COLLATE section. It appears to me that either the output from the locale utility doesn't use the correct range, or that the valid range for the parser in localedef is incorrect.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Assigned: (STDCXX-277) [locale] invalid LC_COLLATE output
for weights > 255
Posted by "Martin Sebor (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/STDCXX-277?page=all ]
Martin Sebor reassigned STDCXX-277:
-----------------------------------
Assignee: Martin Sebor
> [locale] invalid LC_COLLATE output for weights > 255
> ----------------------------------------------------
>
> Key: STDCXX-277
> URL: http://issues.apache.org/jira/browse/STDCXX-277
> Project: C++ Standard Library
> Issue Type: Bug
> Components: Utilities
> Affects Versions: 4.1.2, 4.1.3
> Environment: Multiple, including Linux 2.6.14.5 + gcc 3.2.3
> Reporter: Andrew Black
> Assigned To: Martin Sebor
> Attachments: en_US.ISO-8859-1.sh.out, out.1
>
>
> To test the locales, the run_locale_utils.sh script is used, residing in the etc/config subdirectory of the source directory.
> During this process, the source file in the etc/nls subdirectory is first transformed into a locale database using the localedef utility, then is converted back to a source file using the locale utility. This process is repeated twice more, with each following stage relying on the source file from the previous itteration. The test then concludes by checking that the source files produced in the second and third stage are identical.
> On some platforms, including my local box, this process fails for most locales when the localedef utility in the second stage is unable to process the output of the source file generated by the locale utility in the first stage of the process. Output from running the en_US.ISO-8859-1.sh (wrapper) script with the '-d' flag on my box is attached as en_US.ISO-8859-1.sh.out. The failure message from the localedef utility in this case is as follows:
> Error 399: decimal value in the range [0, 255) expected: \d383
> When I search for the string '\d383' in the source file being parsed (attached as out.1), I find it in the LC_COLLATE section. It appears to me that either the output from the locale utility doesn't use the correct range, or that the valid range for the parser in localedef is incorrect.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (STDCXX-277) [locale] invalid LC_COLLATE output for
weights > 255
Posted by "Martin Sebor (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/STDCXX-277?page=all ]
Martin Sebor updated STDCXX-277:
--------------------------------
Summary: [locale] invalid LC_COLLATE output for weights > 255 (was: Locale stage 2 test failure)
Affects Version/s: 4.1.2
4.1.3
> [locale] invalid LC_COLLATE output for weights > 255
> ----------------------------------------------------
>
> Key: STDCXX-277
> URL: http://issues.apache.org/jira/browse/STDCXX-277
> Project: C++ Standard Library
> Issue Type: Bug
> Components: Utilities
> Affects Versions: 4.1.2, 4.1.3
> Environment: Multiple, including Linux 2.6.14.5 + gcc 3.2.3
> Reporter: Andrew Black
> Attachments: en_US.ISO-8859-1.sh.out, out.1
>
>
> To test the locales, the run_locale_utils.sh script is used, residing in the etc/config subdirectory of the source directory.
> During this process, the source file in the etc/nls subdirectory is first transformed into a locale database using the localedef utility, then is converted back to a source file using the locale utility. This process is repeated twice more, with each following stage relying on the source file from the previous itteration. The test then concludes by checking that the source files produced in the second and third stage are identical.
> On some platforms, including my local box, this process fails for most locales when the localedef utility in the second stage is unable to process the output of the source file generated by the locale utility in the first stage of the process. Output from running the en_US.ISO-8859-1.sh (wrapper) script with the '-d' flag on my box is attached as en_US.ISO-8859-1.sh.out. The failure message from the localedef utility in this case is as follows:
> Error 399: decimal value in the range [0, 255) expected: \d383
> When I search for the string '\d383' in the source file being parsed (attached as out.1), I find it in the LC_COLLATE section. It appears to me that either the output from the locale utility doesn't use the correct range, or that the valid range for the parser in localedef is incorrect.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (STDCXX-277) Locale stage 2 test failure
Posted by "Andrew Black (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/STDCXX-277?page=comments#action_12430284 ]
Andrew Black commented on STDCXX-277:
-------------------------------------
A little more digging into the situation from a slightly different direction.
On my machine, the lone locale that passes the tests is the ar_SA.ISO-8859-6 locale. This leads to a question about what makes this locale unique? One observation is that it defines its own LC_COLLATE table. I would estimate that at least half the locales rely on the table specified in the iso14651_t1 file, either directly or indirectly, based on a search of the locale source files.
After doing some stepping through the locale utility, it appears that the out of range value is being printed in the following stack trace
#0 print_weight(unsigned const*, unsigned, unsigned) (weightp=0xb7ef0988,
num_weights=4, longest_weight=2)
at /build/ablack/stdcxx-test/util/locale.cpp:576
#1 0x0804c05f in write_coll_info(std::string const&, unsigned, unsigned) (
ch=@0xbf9065b0, idx=0, tab_num=0)
at /build/ablack/stdcxx-test/util/locale.cpp:643
#2 0x0804c8a8 in print_lc_collate() ()
at /build/ablack/stdcxx-test/util/locale.cpp:776
#3 0x08054e9d in main (argc=0, argv=0xbf90666c)
at /build/ablack/stdcxx-test/util/locale.cpp:2555
#4 0xb7d4fd06 in __libc_start_main () from /lib/libc.so.6
The first parameter to the print_weight function is the return value from 'collate_st->get_weight (tab[i - first])', where collate_st is an _RW::__rw_collate_t object pointer
> Locale stage 2 test failure
> ---------------------------
>
> Key: STDCXX-277
> URL: http://issues.apache.org/jira/browse/STDCXX-277
> Project: C++ Standard Library
> Issue Type: Bug
> Components: Utilities
> Environment: Multiple, including Linux 2.6.14.5 + gcc 3.2.3
> Reporter: Andrew Black
> Attachments: en_US.ISO-8859-1.sh.out, out.1
>
>
> To test the locales, the run_locale_utils.sh script is used, residing in the etc/config subdirectory of the source directory.
> During this process, the source file in the etc/nls subdirectory is first transformed into a locale database using the localedef utility, then is converted back to a source file using the locale utility. This process is repeated twice more, with each following stage relying on the source file from the previous itteration. The test then concludes by checking that the source files produced in the second and third stage are identical.
> On some platforms, including my local box, this process fails for most locales when the localedef utility in the second stage is unable to process the output of the source file generated by the locale utility in the first stage of the process. Output from running the en_US.ISO-8859-1.sh (wrapper) script with the '-d' flag on my box is attached as en_US.ISO-8859-1.sh.out. The failure message from the localedef utility in this case is as follows:
> Error 399: decimal value in the range [0, 255) expected: \d383
> When I search for the string '\d383' in the source file being parsed (attached as out.1), I find it in the LC_COLLATE section. It appears to me that either the output from the locale utility doesn't use the correct range, or that the valid range for the parser in localedef is incorrect.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira