You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2026/05/03 12:37:05 UTC
[I] Introduce `StringViewArrayBuilder::map` to avoid duplication [datafusion]
alamb opened a new issue, #21997:
URL: https://github.com/apache/datafusion/issues/21997
I think you could avoid a lot of this copy / paste by using a `StringViewBuilder` with the `append_block` and `append_view_unchecked` methods
https://docs.rs/arrow/latest/arrow/array/type.StringViewBuilder.html#method.append_block
That being said, I do think it could be slightly slower than this implementation because it would have to re-check the length
It almost seems like what we want is some sort of API on `StringViewArray` itself, similar to https://docs.rs/arrow/latest/arrow/array/struct.PrimitiveArray.html#method.unary
So this code could be written something like
```rust
let new_array = orig_array.map_values(convert)
```
That would also let us do potentially crazy things like reuse the buffer allocations and modify the values in place if they weren't shared 🤔
If that makes sense to you I can file a ticket in arrow-rs perhaps.
_Originally posted by @alamb in https://github.com/apache/datafusion/pull/21980#discussion_r3178091741_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Introduce `StringViewArrayBuilder::map` to avoid duplication [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #21997: Introduce `StringViewArrayBuilder::map` to avoid duplication
URL: https://github.com/apache/datafusion/issues/21997
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Introduce `StringViewArrayBuilder::map` to avoid duplication [datafusion]
Posted by "neilconway (via GitHub)" <gi...@apache.org>.
neilconway commented on issue #21997:
URL: https://github.com/apache/datafusion/issues/21997#issuecomment-4366702339
@buraksenn I'm already working on a PR for the `append_with` idea I mentioned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Introduce `StringViewArrayBuilder::map` to avoid duplication [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #21997:
URL: https://github.com/apache/datafusion/issues/21997#issuecomment-4366185806
As @neilconway says https://github.com/apache/datafusion/pull/21980#issuecomment-4366173666
> I meant something similar to what you suggested in the PR -- for example, a function like:
```rust
impl StringViewArrayBuilder {
/// Reserve `len` bytes of output storage and let the caller fill them.
pub fn append_with<F: FnOnce(&mut [u8])>(&mut self, len: usize, fill: F);
}
```
> We could use that in `reverse`, `translate`, `initcap`, and `lower`/`upper`. I want to check whether it regresses performance against the hand-optimized version though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Introduce `StringViewArrayBuilder::map` to avoid duplication [datafusion]
Posted by "neilconway (via GitHub)" <gi...@apache.org>.
neilconway commented on issue #21997:
URL: https://github.com/apache/datafusion/issues/21997#issuecomment-4366703510
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Introduce `StringViewArrayBuilder::map` to avoid duplication [datafusion]
Posted by "buraksenn (via GitHub)" <gi...@apache.org>.
buraksenn commented on issue #21997:
URL: https://github.com/apache/datafusion/issues/21997#issuecomment-4366692806
@neilconway will you take a look into this? If not I can take a look into this quickly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Introduce `StringViewArrayBuilder::map` to avoid duplication [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #21997:
URL: https://github.com/apache/datafusion/issues/21997#issuecomment-4366188305
I think the idea of reusing the allocations for StringView is an interesting one (e.g. mutating the bytes in place when possible), especially for our string functions which will often be creating a temporary intermediate (the input is often not used again)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org