You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Kev1n8 (via GitHub)" <gi...@apache.org> on 2024/09/09 07:06:27 UTC
[I] Improve performance of *TRIM functions for StringViewArray [datafusion]
Kev1n8 opened a new issue, #12387:
URL: https://github.com/apache/datafusion/issues/12387
### Is your feature request related to a problem or challenge?
StringView provides us a way to directly modify views of an array instead of generating a new StringArray when it comes to string operations like `substr`, which was done in #12044. I think we can play a similar trick in `trim`.
### Describe the solution you'd like
I've looked into the implementation of [`general_trim`](https://github.com/apache/datafusion/blob/f5c47fa274d53c1d524a1fb788d9a063bf5240ef/datafusion/functions/src/string/common.rs#L51), it uses the `str::trim_xxx_matches` methods the obtain the "substring". Furthermore, inside the method, it first computes the `[start, end)` boundary and slice the `str`. The index here is useful for modifying views. Unfortunately, currently the feature `Pattern` it uses is `unstable`.
### Describe alternatives you've considered
I'm not very sure what alternatives we have. Perhaps either implement our own `trim` or wait until `Pattern` is stable in Rust.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Improve performance of *TRIM functions for StringViewArray [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on issue #12387:
URL: https://github.com/apache/datafusion/issues/12387#issuecomment-2337672517
Seems we can get the index through this way:
- Convert the originanl `str1` to `ptr1(*const [u8])` first.
- Then we use `trim_xxx_matches` first to get the `str2`.
- Convert the `str2` to `ptr2(*cosnt [u8])`, and we get the index through `ptr1 - ptr2`?
Maybe I can try it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Improve performance of *TRIM functions for StringViewArray [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on issue #12387:
URL: https://github.com/apache/datafusion/issues/12387#issuecomment-2337937005
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Improve performance of *TRIM functions for StringViewArray [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #12387: Improve performance of *TRIM functions for StringViewArray
URL: https://github.com/apache/datafusion/issues/12387
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Improve performance of *TRIM functions for StringViewArray [datafusion]
Posted by "Kev1n8 (via GitHub)" <gi...@apache.org>.
Kev1n8 commented on issue #12387:
URL: https://github.com/apache/datafusion/issues/12387#issuecomment-2337933460
Nice great! @Rachelint
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Improve performance of *TRIM functions for StringViewArray [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on issue #12387:
URL: https://github.com/apache/datafusion/issues/12387#issuecomment-2374635119
Finally, after reminding from @alamb , I found we can get the index in safe way, detail can see:
https://github.com/Rachelint/arrow-datafusion/blob/f8174626e47d147e90c6715f5052ccfa269f0493/datafusion/functions/src/string/common.rs#L80
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org