You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2024/06/14 19:31:06 UTC
[I] Implement equality `=` and inequality `<>` support for `StringView` [datafusion]
alamb opened a new issue, #10919:
URL: https://github.com/apache/datafusion/issues/10919
### Is your feature request related to a problem or challenge?
Part of https://github.com/apache/datafusion/issues/10918, `[StringViewArray`](https://docs.rs/arrow/latest/arrow/array/type.StringViewArray.html) support in DataFusion
There are several queries in the [clickbench](https://github.com/apache/datafusion/blob/11b7b5c215012231e5768fc5be3445c0254d0169/benchmarks/queries/clickbench/queries.sql#L11-L15) suite like follows:
```sql
SELECT "MobilePhone", "MobilePhoneModel", COUNT(DISTINCT "UserID") AS u FROM hits WHERE "MobilePhoneModel" <> '' GROUP BY "MobilePhone", "MobilePhoneModel" ORDER BY u DESC LIMIT 10;
SELECT "SearchPhrase", COUNT(*) AS c FROM hits WHERE "SearchPhrase" <> '' GROUP BY "SearchPhrase" ORDER BY c DESC LIMIT 10;
SELECT "SearchPhrase", COUNT(DISTINCT "UserID") AS u FROM hits WHERE "SearchPhrase" <> '' GROUP BY "SearchPhrase" ORDER BY u DESC LIMIT 10;
SELECT "SearchEngineID", "SearchPhrase", COUNT(*) AS c FROM hits WHERE "SearchPhrase" <> '' GROUP BY "SearchEngineID", "SearchPhrase" ORDER BY c DESC LIMIT 10;
```
where `"MobilePhoneModel"` and `"SearchPhrase"` are string columns with predicates (in this case checking for empty string)
### Describe the solution you'd like
In order to improve performance of these queries we will need the ability to actually compare `StringViewArrays` to constant strings (and likely to each other)
Thus I would like to be able to run
`StringViewColumn = scalar`
`StringViewColumn = StringViewColumn`
(and likewise for BinaryView)
I basically want to to run the following queries (where table `foo` has `StringView` columns)
```sql
> create table foo as values ('Andrew', 'X'), ('Xiangpeng', 'Xiangpeng'), ('Raphael', 'R');
0 row(s) fetched.
Elapsed 0.002 seconds.
> select * from foo where column1 = 'Andrew';
+---------+---------+
| column1 | column2 |
+---------+---------+
| Andrew | X |
+---------+---------+
1 row(s) fetched.
Elapsed 0.003 seconds.
> select * from foo where column1 <> 'Andrew';
+-----------+-----------+
| column1 | column2 |
+-----------+-----------+
| Xiangpeng | Xiangpeng |
| Raphael | R |
+-----------+-----------+
2 row(s) fetched.
Elapsed 0.001 seconds.
> select * from foo where column1 = column2;
+-----------+-----------+
| column1 | column2 |
+-----------+-----------+
| Xiangpeng | Xiangpeng |
+-----------+-----------+
1 row(s) fetched.
Elapsed 0.002 seconds.
> select * from foo where column1 <> column2;
+---------+---------+
| column1 | column2 |
+---------+---------+
| Andrew | X |
| Raphael | R |
+---------+---------+
2 row(s) fetched.
Elapsed 0.001 seconds.
```
### Describe alternatives you've considered
I suspect we will need to update the coercion logic and maybe also the arrow equality kernels like https://docs.rs/arrow/latest/arrow/compute/kernels/cmp/fn.eq.html
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement equality `=` and inequality `<>` support for `StringView` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #10919:
URL: https://github.com/apache/datafusion/issues/10919#issuecomment-2170257967
> This issue must wait until #10920 because there is currently no convenient way to create a `StringViewArray` in Datafusion. If I am mistaken, please correct me.
I think you are right -- conveniently @XiangpengHao has one here https://github.com/apache/datafusion/pull/10925
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement equality `=` and inequality `<>` support for `StringView` [datafusion]
Posted by "Weijun-H (via GitHub)" <gi...@apache.org>.
Weijun-H commented on issue #10919:
URL: https://github.com/apache/datafusion/issues/10919#issuecomment-2169307386
This issue must wait until #10920 because there is currently no convenient way to create a `StringViewArray` in Datafusion. If I am mistaken, please correct me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement equality `=` and inequality `<>` support for `StringView` [datafusion]
Posted by "XiangpengHao (via GitHub)" <gi...@apache.org>.
XiangpengHao commented on issue #10919:
URL: https://github.com/apache/datafusion/issues/10919#issuecomment-2170965292
Hi @Weijun-H , great to know you are working on this!
I believe implementing this feature will eventually require https://github.com/apache/arrow-rs/issues/5897 to be solved, so I'm working on that issue so you won't be blocked
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement equality `=` and inequality `<>` support for `StringView` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #10919:
URL: https://github.com/apache/datafusion/issues/10919#issuecomment-2174349963
BTW I made a branch to work on StringView in DataFusion: https://github.com/apache/datafusion/issues/10961
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Implement equality `=` and inequality `<>` support for `StringView` [datafusion]
Posted by "Weijun-H (via GitHub)" <gi...@apache.org>.
Weijun-H commented on issue #10919:
URL: https://github.com/apache/datafusion/issues/10919#issuecomment-2169073038
I am glad to pick this ticket.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org