You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2024/09/05 12:52:22 UTC
[I] Use `StringViewArray` as output of `substr` when input was `StringArray` [datafusion]
alamb opened a new issue, #12338:
URL: https://github.com/apache/datafusion/issues/12338
### Is your feature request related to a problem or challenge?
Part of https://github.com/apache/datafusion/issues/11752
StringView is a new arrow array type that allows for more efficient string processing -- specifically it allows string data to be adjusted without copying the underlying data
See this blog post for more details: https://www.influxdata.com/blog/faster-queries-with-stringview-part-one-influxdb/
@Kev1n8 added support for `StringView` to the `substr` function in https://github.com/apache/datafusion/pull/12044
At the moment `substr` produces a `StringArray` output when the input is `StringArray`, but we could actually generate a `StringViewArray` as output which would be more efficient in most cases (avoids copying the string values)
However, in order to avoid errors when `substr` is used in an expression, we need to make sure that all the rest of the String functions support StringView as input as well. Aka we should wait for the "Required for enabling StringView by default" list on https://github.com/apache/datafusion/issues/11752 to be completed
### Describe the solution you'd like
1. change the output type of `substr` to be `StringViewArray` when the input is `StringArray` (note for `LargeStringArray` we will still need to copy the data I think as `StringView` is limited to 2^32 bytes)
2. Change the implementation of `substr` to use `StringView` internally
3. Add tests
### Describe alternatives you've considered
_No response_
### Additional context
Note that @Kevin8 has already added support for `StringView` to the `substr` function in https://github.com/apache/datafusion/pull/12044
They also suggested this same optimization could be applied https://github.com/apache/datafusion/pull/12044#issuecomment-2316111793
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Use `StringViewArray` as output of `substr` when input was `StringArray` [datafusion]
Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
2010YOUY01 closed issue #12338: Use `StringViewArray` as output of `substr` when input was `StringArray`
URL: https://github.com/apache/datafusion/issues/12338
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Use `StringViewArray` as output of `substr` when input was `StringArray` [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on issue #12338:
URL: https://github.com/apache/datafusion/issues/12338#issuecomment-2334694805
I would propose that this change when made happens only after https://github.com/apache/datafusion/issues/12119 lands.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Use `StringViewArray` as output of `substr` when input was `StringArray` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #12338:
URL: https://github.com/apache/datafusion/issues/12338#issuecomment-2491402048
> Should this be the default for all functions that handle strings now? Require they can accept Utf8View as input for all fields and produce Utf8View output ?
I am not sure
I think each function should produce Utf8 / Utf8View depending on what makes sense (Utf8 is more efficient if you have to rewrite all the strings anyways)
Once string functions accept either Utf8/Utf8View which I think would avoid the chained casting you are describing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Use `StringViewArray` as output of `substr` when input was `StringArray` [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on issue #12338:
URL: https://github.com/apache/datafusion/issues/12338#issuecomment-2491341800
Should this be the default for all functions that handle strings now? Require they can accept Utf8View as input for all fields and produce Utf8View output ?
I ask because that should solve issues with chained functions causing a series of casts such as
<function 1 = utf8View -> Ut8), <function 2 = utf8 -> Ut8) ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Use `StringViewArray` as output of `substr` when input was `StringArray` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #12338:
URL: https://github.com/apache/datafusion/issues/12338#issuecomment-2491312244
This is probably a good exercise now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org