You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "jonathanc-n (via GitHub)" <gi...@apache.org> on 2024/11/10 06:24:56 UTC
[PR] feat: Add `stringview` for functions to prevent regression [datafusion]
jonathanc-n opened a new pull request, #13332:
URL: https://github.com/apache/datafusion/pull/13332
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
Closes #13330 .
## Rationale for this change
Add string view to some functions to prevent regression during @timsaucer df python migration.
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
Added support for utf8view to all listed functions in the issue except for `ConcatFunc`
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
## Are these changes tested?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add `stringview` for functions to prevent regression [datafusion]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on PR #13332:
URL: https://github.com/apache/datafusion/pull/13332#issuecomment-2466735250
Wonderful. Thank you. I'll try to review tomorrow.
I did dive into the ConcatFunc and I found (and resolved) two issues:
1) when doing concat on pure literals we aren't allowing for utf8view or large utf8view with values. This isn't caught in the unit tests because they don't have these variants
2) `simplify_concat` is changing the merged types to utf8, which is what caused the schema error.
I'll put up the PR today or tomorrow to fix both of these, with unit tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add `stringview` for functions to prevent regression [datafusion]
Posted by "jonathanc-n (via GitHub)" <gi...@apache.org>.
jonathanc-n commented on PR #13332:
URL: https://github.com/apache/datafusion/pull/13332#issuecomment-2474968722
@timsaucer Will push some tests, however after going through it, `date_bin` need to wait on this conversion to work:
`Unsupported CAST from Timestamp(Nanosecond, None) to Utf8View` (a issue can be opened for this in arrow-rs to do this as a double cast is probably not ideal). `named_struct` also has this problem, if you want the output to be defaulted to be utf8view this has to be solved in arrow for `array_struct`. Even if i defined the schema in datafusion to output Utf8View or input Utf8View, there will be a schema mismatch or a conversion error, due to the data originally being Utf8.
I changed this pr to only deal with support stringview for `array_to_string`, `encoding`, and `bit_length`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add `stringview` for functions to prevent regression [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on PR #13332:
URL: https://github.com/apache/datafusion/pull/13332#issuecomment-2475156658
Update array_to_string and string_to_array in a [separate PR](https://github.com/apache/datafusion/pull/13403) - it's a bit more extensive than what this PR contains.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add `stringview` for functions to prevent regression [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #13332:
URL: https://github.com/apache/datafusion/pull/13332#issuecomment-2473680628
Thank you @jonathanc-n -- I marked this PR as draft as you seem to be planning to add more tests. Please mark it as ready for review with it is ready for another look
Thanks a lot for sorting this out
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add `stringview` for functions to prevent regression [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on PR #13332:
URL: https://github.com/apache/datafusion/pull/13332#issuecomment-2477150667
> > @timsaucer Will push some tests, however after going through it, `date_bin` need to wait on this conversion to work: `Unsupported CAST from Timestamp(Nanosecond, None) to Utf8View` (a issue can be opened for this in arrow-rs to do this as a double cast is probably not ideal).
>
> Is this an arrow-rs issue or is this #13363 ?
Possibly both. I think arrow-cast is missing the _,Utf8View for primitive & temporal casting here: https://github.com/apache/arrow-rs/blob/1d580ec1996e75bb9be4c1c880871d0dd2718ad4/arrow-cast/src/cast/mod.rs#L1465
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add `stringview` for functions to prevent regression [datafusion]
Posted by "jonathanc-n (via GitHub)" <gi...@apache.org>.
jonathanc-n commented on PR #13332:
URL: https://github.com/apache/datafusion/pull/13332#issuecomment-2466607191
@timsaucer Do you think testing is necessary here? Can add if so. Also, are you able to show how to replicate the optimizer error for `ConcatFunc`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add `stringview` for functions to prevent regression [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on PR #13332:
URL: https://github.com/apache/datafusion/pull/13332#issuecomment-2477127805
> @timsaucer Will push some tests, however after going through it, `date_bin` need to wait on this conversion to work: `Unsupported CAST from Timestamp(Nanosecond, None) to Utf8View` (a issue can be opened for this in arrow-rs to do this as a double cast is probably not ideal).
Is this an arrow-rs issue or is this https://github.com/apache/datafusion/issues/13363 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add `stringview` for functions to prevent regression [datafusion]
Posted by "jonathanc-n (via GitHub)" <gi...@apache.org>.
jonathanc-n commented on PR #13332:
URL: https://github.com/apache/datafusion/pull/13332#issuecomment-2469509999
Will add tests tomorrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add `stringview` for functions to prevent regression [datafusion]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer commented on PR #13332:
URL: https://github.com/apache/datafusion/pull/13332#issuecomment-2466789423
ConcatFunc: https://github.com/apache/datafusion/pull/13346
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add `stringview` for functions to prevent regression [datafusion]
Posted by "jonathanc-n (via GitHub)" <gi...@apache.org>.
jonathanc-n commented on PR #13332:
URL: https://github.com/apache/datafusion/pull/13332#issuecomment-2475181836
@Omega359 Should i remove my changes for array_to_string?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add `stringview` support to `encode` and `decode` and `bit_length` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #13332:
URL: https://github.com/apache/datafusion/pull/13332#issuecomment-2481173925
Thanks again @jonathanc-n
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add `stringview` support to `encode` and `decode` and `bit_length` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #13332:
URL: https://github.com/apache/datafusion/pull/13332
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add `stringview` support to `encode` and `decode` and `bit_length` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #13332:
URL: https://github.com/apache/datafusion/pull/13332#discussion_r1844306243
##########
datafusion/functions/src/encoding/inner.rs:
##########
@@ -224,6 +224,7 @@ fn encode_process(value: &ColumnarValue, encoding: Encoding) -> Result<ColumnarV
ColumnarValue::Array(a) => match a.data_type() {
DataType::Utf8 => encoding.encode_utf8_array::<i32>(a.as_ref()),
DataType::LargeUtf8 => encoding.encode_utf8_array::<i64>(a.as_ref()),
+ DataType::Utf8View => encoding.encode_utf8_array::<i32>(a.as_ref()),
Review Comment:
I think this means the output type will be Utf8 but for this type i think that is totally fine (there is no need to preserve the view encoding)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org