You are viewing a plain text version of this content. The canonical link for it is here.

Posted to jira@arrow.apache.org by "Mike Seddon (Jira)" <ji...@apache.org> on 2021/01/30 00:40:00 UTC

[jira] [Closed] (ARROW-11339) [Rust][DataFusion] length kernel does not correctly calculate character length

     [ https://issues.apache.org/jira/browse/ARROW-11339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mike Seddon closed ARROW-11339.
-------------------------------
    Resolution: Duplicate

https://issues.apache.org/jira/browse/ARROW-11434

> [Rust][DataFusion] length kernel does not correctly calculate character length
> ------------------------------------------------------------------------------
>
>                 Key: ARROW-11339
>                 URL: https://issues.apache.org/jira/browse/ARROW-11339
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: Rust - DataFusion
>            Reporter: Mike Seddon
>            Assignee: Mike Seddon
>            Priority: Minor
>
> The current kernel works for simple characters as it appears to be assuming that 1 byte = 1 character. this is very fast but is not a safe assumption given Arrow strings are utf8. 
> A simple example of failure is from the Postgres example where the current `length` implementation will calculate 5.
> `char_length('josé') → 4`
> The correct method seems to be via https://docs.rs/unicode-segmentation/1.2.1/unicode_segmentation/struct.Graphemes.html which I can implement in my work here: https://github.com/apache/arrow/pull/9243 and remove from kernel.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)