[jira] [Commented] (IMPALA-10761) Provide query option for illegal UTF-8 characters handling

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-10761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17384390#comment-17384390 ] 

ASF subversion and git services commented on IMPALA-10761:
----------------------------------------------------------

Commit 4df03a31ec77b54138aba2805ff5e376463c8e23 in impala's branch refs/heads/master from stiga-huang
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=4df03a3 ]

IMPALA-2019(Part-2): Provide UTF-8 support in instr() and locate()

Similar to the previous patch, this patch adds UTF-8 support in instr()
and locate() builtin functions so they can have consistent behaviors
with Hive's. These two string functions both have an optional argument
as position:
INSTR(STRING str, STRING substr[, BIGINT position[, BIGINT occurrence]])
LOCATE(STRING substr, STRING str[, INT pos])
Their return values are positions of the matched substring.

In UTF-8 mode (turned on by set UTF8_MODE=true), these positions are
counted by UTF-8 characters instead of bytes.

Error handling:
Malformed UTF-8 characters are counted as one byte per character. This
is consistent with Hive since Hive replaces those bytes to U+FFFD
(REPLACEMENT CHARACTER). E.g. GenericUDFInstr calls Text#toString(),
which performs the replacement. We can provide more behaviors on error
handling like ignoring them or reporting errors. IMPALA-10761 will focus
on this.

Tests:
 - Add BE unit tests and e2e tests
 - Add random tests to make sure malformed UTF-8 characters won't crash
   us.

Change-Id: Ic13c3d04649c1aea56c1aaa464799b5e4674f662
Reviewed-on: http://gerrit.cloudera.org:8080/17580
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>


> Provide query option for illegal UTF-8 characters handling
> ----------------------------------------------------------
>
>                 Key: IMPALA-10761
>                 URL: https://issues.apache.org/jira/browse/IMPALA-10761
>             Project: IMPALA
>          Issue Type: New Feature
>            Reporter: Quanlong Huang
>            Priority: Major
>
> There are 3 ways to handle illegal UTF-8 characters:
>  * Replacing them with U+FFFD (REPLACEMENT CHARACTER)
>  * Ignoring them (removing them in the string)
>  * Reporting errors
> We should provide a query option for this.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org