Re: [D] How to handle `null` values in a UDAF? [datafusion]

["rspears74 (via GitHub)" <gi...@apache.org>]

GitHub user rspears74 closed a discussion: How to handle `null` values in a UDAF?

I am trying to handle null values in the `update_batch` method of an `Accumulator` when defining a UDAF. I've been trying something like the following:
```
fn update_batch(&mut self, values: &[ArrayRef]) -> Result<()> {
    let vals: Vec<String> = (&values[0])
        .as_string::<i32>()
        .iter()
        .flatten()
        .map(|s| String::from(s))
        .collect();

    self.merge_values(vals);
    Ok(())
}
```
... but I get 
```
thread 'tokio-runtime-worker' panicked at /.../.cargo/registry/src/index.crates.io-6f17d22bba15001f/arrow-data-49.0.0/src/transform/list.rs:36:69:
range end index 2 out of range for slice of length 1
```

I was able to make it work for numeric types by sort of mimicking what some of the internal aggregators do:
```
fn update_batch(&mut self, values: &[ArrayRef]) -> Result<()> {
    let vals = (&values[0])
        .as_primitive::<Float64Type>()
        .values()
        .to_vec();

    self.merge_values(vals);
    Ok(())
}
```
... but this doesn't seem possible with Strings since calling `values` on a `StringArray` returns a `Vec<u8>` instead of a `Vec<&str>` (or `String`). Is there something I'm doing wrong in the first example where I'm using `flatten`? That should flatten a list of `[Some("string"), Some("other string"), None]` to `["string", "other string"]`, but I suppose there's something going on internally that's not allowing that to work.

GitHub link: https://github.com/apache/datafusion/discussions/8974

----
This is an automatically sent email for github@datafusion.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@datafusion.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org