You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Antoine Pitrou <an...@python.org> on 2023/07/02 12:07:20 UTC
Re: [DISCUSS] UTF-8 validation
Le 02/07/2023 à 14:00, Raphael Taylor-Davies a écrit :
>
> More an observation than an issue, but UTF-8 validation for StringArray can be done very efficiently by first verifying the entire buffer, and then verifying the offsets correspond to the start of a UTF-8 codepoint.
Caveat: null slots could potentially contain invalid UTF-8 data. Not
likely of course, but it should probably not be an error.
That said, yes, it is a smart strategy for the common case!
Regards
Antoine.
Re: [DISCUSS] UTF-8 validation
Posted by Raphael Taylor-Davies <r....@googlemail.com.INVALID>.
For better or for worse the Rust implementation requires the underlying buffer is UTF-8 including null slots, as this allows returning the buffer as a native string type, which in turn allows kernels to use Rust's native string functionality. Whilst I agree the specification is ambiguous on this note, this interpretation doesn't appear to have caused issues so far.
On 2 July 2023 13:07:20 BST, Antoine Pitrou <an...@python.org> wrote:
>
>
>Le 02/07/2023 à 14:00, Raphael Taylor-Davies a écrit :
>>
>> More an observation than an issue, but UTF-8 validation for StringArray can be done very efficiently by first verifying the entire buffer, and then verifying the offsets correspond to the start of a UTF-8 codepoint.
>
>Caveat: null slots could potentially contain invalid UTF-8 data. Not likely of course, but it should probably not be an error.
>
>That said, yes, it is a smart strategy for the common case!
>
>Regards
>
>Antoine.