You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "stuartcarnie (via GitHub)" <gi...@apache.org> on 2023/02/01 20:45:55 UTC
[GitHub] [arrow-datafusion] stuartcarnie opened a new pull request, #5152: feat: Type coercion for Dictionary(_, _) to Utf8 for regex conditions
stuartcarnie opened a new pull request, #5152:
URL: https://github.com/apache/arrow-datafusion/pull/5152
# Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
Closes #.
# Rationale for this change
This change teaches DataFusion how to coerce Dictionary(_, string-like-type) to a string-like type for regex conditional expressions, so the comparison may succeed.
# What changes are included in this PR?
Updated the type coercion rules when encountering one of the regular expression operators.
# Are these changes tested?
Yes, new tests were added to validate the changes.
# Are there any user-facing changes?
Queries that previously would fail with an error that the Dictionary type is incompatible with a regular expression condition will now succeed. IOx uses `Dictionary(Int32, Utf8)` columns for tags.
In some cases, regular expression queries succeed, such as:
```sql
SELECT usage_idle FROM cpu WHERE cpu ~ '9'
```
As the optimiser changes the filter plan from a regex conditional to a `LIKE '%9%'`, and the `LIKE` operator has additional code to coerce dictionary types:
https://github.com/apache/arrow-datafusion/blob/031534d94efb305eb26a7c16fd7e06ae3bcd88bb/datafusion/expr/src/type_coercion/binary.rs#L522
however, other cases unexpectedly fail:
```sql
SELECT usage_idle FROM cpu WHERE cpu ~ '9$'
```
as the optimiser does not rewrite this to a `LIKE` expression[^1]
[^1]: Incidentally, the optimiser could also rewrite this to `LIKE '%9'`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] ursabot commented on pull request #5152: feat: Type coercion for Dictionary(_, _) to Utf8 for regex conditions
Posted by "ursabot (via GitHub)" <gi...@apache.org>.
ursabot commented on PR #5152:
URL: https://github.com/apache/arrow-datafusion/pull/5152#issuecomment-1416413134
Benchmark runs are scheduled for baseline = e69cd559b78ec58ed7fe689e928dbdd0e976bac3 and contender = 100665c482c2fd3651e6446547e28b18e057f7cd. 100665c482c2fd3651e6446547e28b18e057f7cd is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ec2-t3-xlarge-us-east-2] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/4f7db9aca1bf4c5b9ffc03f6f7407e30...a560bc27ae854608ac731906e3cf1651/)
[Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on test-mac-arm] [test-mac-arm](https://conbench.ursa.dev/compare/runs/7b70d1d9b0714d2a8168dbb5359c8d09...f9e6762dcacf462daefbcda8140a6424/)
[Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ursa-i9-9960x] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/133617ef571b4d45a4a09438955326e6...f1c355ad69f64a2d85af2d99db059606/)
[Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ursa-thinkcentre-m75q] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/27e9fdd3da96418c9a7beb32cd3bb06f...cd6f73031be24b7d801a0cdb6e5ecf64/)
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb merged pull request #5152: feat: Type coercion for Dictionary(_, _) to Utf8 for regex conditions
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #5152:
URL: https://github.com/apache/arrow-datafusion/pull/5152
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org