[I] `substring` errors when the bytes under a null `Utf8` slot would be cut inside a character [arrow-rs]

["mbutrovich (via GitHub)" <gi...@apache.org>]

mbutrovich opened a new issue, #11590:
URL: https://github.com/apache/arrow-rs/issues/11590

   ### Describe the bug
   
   `substring` on a `StringArray` or `LargeStringArray` checks char boundaries in every slot, including null slots. If a null slot covers a multi-byte character and the cut falls inside it, the whole call returns an error, although the output for that slot is null.
   
   ### To Reproduce
   
   ```rust
   use arrow_array::{BooleanArray, StringArray};
   use arrow_select::nullif::nullif;
   use arrow_string::substring::substring;
   
   fn main() {
       let array = nullif(&StringArray::from(vec!["\u{e9}", "ab"]), &BooleanArray::from(vec![true, false])).unwrap();
       println!("{:?}", substring(&array, 1, None));
       // Err(ComputeError("The offset 1 is at an invalid utf-8 boundary."))
   }
   ```
   
   ### Expected behavior
   
   `Ok` with `[null, "b"]`. Only non-null values should be able to cause the documented char boundary error.
   
   ### Additional context
   
   The [Columnar format](https://github.com/apache/arrow/blob/fcf8f3014227cafe24deeb044c2d51f5fc9902e5/docs/source/format/Columnar.rst?plain=1#L442-L445) lets a null slot cover a non-empty range with undefined contents. `byte_substring` computes the new start and end of every slot through `check_char_boundary`, with no null check ([substring.rs](https://github.com/apache/arrow-rs/blob/72887ed91c80002bb7ec755ce1e964bab07bebea/arrow-string/src/substring.rs#L379-L400)). `string_view_substring` skips null slots before the same check ([substring.rs](https://github.com/apache/arrow-rs/blob/72887ed91c80002bb7ec755ce1e964bab07bebea/arrow-string/src/substring.rs#L287-L300)), so a `StringViewArray` with a null view over the same bytes returns `[null, "b"]`.
   
   `nullif` keeps the bytes of the slots it sets to null, so its output has this shape. Arrays from other producers through FFI or IPC can too.
   
   Found with property-based tests that build arrays in every physical layout the Arrow format allows.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] `substring` errors when the bytes under a null `Utf8` slot would be cut inside a character [arrow-rs]

["cakeni (via GitHub)" <gi...@apache.org>]

cakeni commented on issue #11590:
URL: https://github.com/apache/arrow-rs/issues/11590#issuecomment-6235462319

   take


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org