[jira] [Commented] (AVRO-1593) C++ json encoder assumes "C" locale and generates invalid UTF-8 sequence

["Hatem Helal (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AVRO-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14166625#comment-14166625 ] 

Hatem Helal commented on AVRO-1593:
-----------------------------------

JsonGenerator::doEncodeString relies on calling "iscntrl" to determine whether a given byte is a control character.  In the windows-1252 code page the byte "\x81" is a control character but not in the C locale which breaks the UTF-8 byte sequence.  

A workaround is to temporarily set the LC_CTYPE locale to "C", do the json encoding, and then reset the LC_CTYPE back to its previous value but modifying the global locale settings isn't thread safe.  A better approach would be to use the newer "std::iscntrl" function which accepts a locale object.  A patch is available here:

http://git.io/XetN-w

> C++ json encoder assumes "C" locale and generates invalid UTF-8 sequence 
> -------------------------------------------------------------------------
>
>                 Key: AVRO-1593
>                 URL: https://issues.apache.org/jira/browse/AVRO-1593
>             Project: Avro
>          Issue Type: Bug
>          Components: c++
>         Environment: windows-1252 encoding
>            Reporter: Hatem Helal
>            Priority: Critical
>
> encoding a multibyte UTF-8 code point such as:
> "\xEF\xBD\x81"
> Incorrectly becomes:
> "\xEF\xBD\U0081"
> When encoded in the service running in the windows-1252 locale.  This isn¹t a valid UTF-8 sequence so we end up with Mojibake when reading back the JSON encoded string.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)