You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Joe McDonnell (Jira)" <ji...@apache.org> on 2023/03/06 20:43:00 UTC

[jira] [Created] (IMPALA-11978) Implement Unicode sandwich for python code

Joe McDonnell created IMPALA-11978:
--------------------------------------

             Summary: Implement Unicode sandwich for python code
                 Key: IMPALA-11978
                 URL: https://issues.apache.org/jira/browse/IMPALA-11978
             Project: IMPALA
          Issue Type: Sub-task
          Components: Infrastructure
    Affects Versions: Impala 4.3.0
            Reporter: Joe McDonnell


Python 3 makes a clear distinction between bytes and strings (Unicode). To handle this appropriately, various places need to be clear about whether they are working on Unicode strings or bytes.

The typical way to fix this for text is to implement a "Unicode sandwich" where the input path is converted to Unicode as early as possible and the output path is converted to bytes as late as possible. This leaves all internal code working on Unicode strings.

Some parts of our code deal with bytes directly (e.g. tests/util/get_parquet_metadata.py has code that deals with the bytes of a Parquet file). Almost everything else should be dealing with Unicode strings.

This is also a good time to fix warnings about the unicode() builtin and basestring.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)