You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "Dandandan (via GitHub)" <gi...@apache.org> on 2025/01/24 10:46:58 UTC
[I] Support using simdutf8 for `validate_string_view` and other utf8 validation [arrow-rs]
Dandandan opened a new issue, #7014:
URL: https://github.com/apache/arrow-rs/issues/7014
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
<!--
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
(This section helps Arrow developers understand the context and *why* for this feature, in addition to the *what*)
-->
We have some different sources were `std::str::from_utf8` is used for utf8 validation, such as `validate_string_view`.
We can speed it up by supporting `simdutf8` as well here.
**Describe the solution you'd like**
<!--
A clear and concise description of what you want to happen.
-->
Similar for https://github.com/apache/arrow-rs/pull/6668 , optionally switch to using `simdutf8` instead.
**Describe alternatives you've considered**
<!--
A clear and concise description of any alternative solutions or features you've considered.
-->
**Additional context**
<!--
Add any other context or screenshots about the feature request here.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support using simdutf8 for `validate_string_view` and other utf8 validation [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #7014:
URL: https://github.com/apache/arrow-rs/issues/7014#issuecomment-2613406009
I am embarassed to admit I missed the StringView one -- I will make a PR to switch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support using simdutf8 for `validate_string_view` and other utf8 validation [arrow-rs]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on issue #7014:
URL: https://github.com/apache/arrow-rs/issues/7014#issuecomment-2623843862
There are places that still do `std::str::from_utf8` like `validate_string_view`.
https://github.com/apache/arrow-rs/blob/3bf29a2c7474e59722d885cd11fafd0dca13a28e/arrow-data/src/byte_view.rs#L145
This still has `std::str::from_utf8` ?
I got it from @alamb profile in https://github.com/apache/datafusion/issues/13983#issuecomment-2613378310 where it has ~5% of samples.
I think there is also some other places which can benefit by a bit from this (e.g. reading csv/json, casting...).
https://github.com/search?q=repo%3Aapache%2Farrow-rs%20std%3A%3Astr%3A%3Afrom_utf8&type=code
> I am embarassed to admit I missed the StringView one -- I will make a PR to switch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support using simdutf8 for `validate_string_view` and other utf8 validation [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on issue #7014:
URL: https://github.com/apache/arrow-rs/issues/7014#issuecomment-2613251782
I mentioned this in #6668, but I was thinking we could just globally replace calls to `from_utf8` with an internal version. Something like
```rust
#[cfg(feature = "simdutf8")]
#[inline(always)]
pub fn from_utf8(val: &[u8]) -> Result<&str, simdutf8::compat::Utf8Error> {
match simdutf8::basic::from_utf8(val) {
Ok(result) => Ok(result),
Err(_) => simdutf8::compat::from_utf8(val),
}
}
#[cfg(not(feature = "simdutf8"))]
#[inline(always)]
pub fn from_utf8(val: &[u8]) -> Result<&str, std::str::Utf8Error> {
std::str::from_utf8(val)
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support using simdutf8 for `validate_string_view` and other utf8 validation [arrow-rs]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on issue #7014:
URL: https://github.com/apache/arrow-rs/issues/7014#issuecomment-2623856307
So I think we can follow the idea of @etseidl to replace all `std::str::from_utf8` calls.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support using simdutf8 for `validate_string_view` and other utf8 validation [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #7014:
URL: https://github.com/apache/arrow-rs/issues/7014#issuecomment-2613414678
Update I don't think I was going crazy. I can't find any remaining location that does utf8 validation that I think will make any different for performance
- https://github.com/apache/arrow-rs/pull/6921#discussion_r1929251692
To be clear, I don't think @etseidl 's idea is a bad one, just I don't think it will help much
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support using simdutf8 for `validate_string_view` and other utf8 validation [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #7014:
URL: https://github.com/apache/arrow-rs/issues/7014#issuecomment-2613427392
Maybe I should have used the positive rather than double negative: " @etseidl 's idea is a good one, I just don't think it will help much " 🤦
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Support using simdutf8 for `validate_string_view` and other utf8 validation [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on issue #7014:
URL: https://github.com/apache/arrow-rs/issues/7014#issuecomment-2613423403
> I don't think it will help much
That is entirely possible. I'm still testing to see if I have any use cases that would benefit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org