You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/03/25 10:36:23 UTC
[GitHub] [arrow-datafusion] yjshen opened a new issue #2093: Add an `InSet` function as an optimized version for `IN`
yjshen opened a new issue #2093:
URL: https://github.com/apache/arrow-datafusion/issues/2093
TPC-DS has many queries with `IN` predicates where all elements are constants. It's a low-hanging fruit if we could implement an `InSet` function for this all constants value case.
While implementing this, we could either use a hashtable or a chain of if-elif-else, depending on the length and the type of the constants array.
Q8:
```SQL
WHERE substr(ca_zip, 1, 5) IN (
'24128','76232','65084','87816','83926','77556','20548',
'26231','43848','15126','91137','61265','98294','25782',
'17920','18426','98235','40081','84093','28577','55565',
'17183','54601','67897','22752','86284','18376','38607',
'45200','21756','29741','96765','23932','89360','29839',
......
```
Q53:
``` SQL
WHERE ss_item_sk = i_item_sk AND
ss_sold_date_sk = d_date_sk AND
ss_store_sk = s_store_sk AND
d_month_seq IN (1200, 1200 + 1, 1200 + 2, 1200 + 3, 1200 + 4, 1200 + 5, 1200 + 6,
1200 + 7, 1200 + 8, 1200 + 9, 1200 + 10, 1200 + 11) AND
((i_category IN ('Books', 'Children', 'Electronics') AND
i_class IN ('personal', 'portable', 'reference', 'self-help') AND
i_brand IN ('scholaramalgamalg #14', 'scholaramalgamalg #7',
'exportiunivamalg #9', 'scholaramalgamalg #9'))
OR
(i_category IN ('Women', 'Music', 'Men') AND
i_class IN ('accessories', 'classical', 'fragrances', 'pants') AND
i_brand IN ('amalgimporto #1', 'edu packscholar #1', 'exportiimporto #1',
'importoamalg #1')))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Ted-Jiang commented on issue #2093: Add an `InSet` function as an optimized version for `IN`
Posted by GitBox <gi...@apache.org>.
Ted-Jiang commented on issue #2093:
URL: https://github.com/apache/arrow-datafusion/issues/2093#issuecomment-1080310241
@yjshen Nice idea! Is this like implement `physical-exr` called `in_set` like existing `in_list`？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] yjshen commented on issue #2093: Add an `InSet` function as an optimized version for `IN`
Posted by GitBox <gi...@apache.org>.
yjshen commented on issue #2093:
URL: https://github.com/apache/arrow-datafusion/issues/2093#issuecomment-1080837823
@Ted-Jiang Another plausible or even preferable way is: we can branch in `in_set` to provide a fast-pass according to its args and reduce the burden it may take to maintain a separate function with a similar semantic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] yjshen edited a comment on issue #2093: Add an `InSet` function as an optimized version for `IN`
Posted by GitBox <gi...@apache.org>.
yjshen edited a comment on issue #2093:
URL: https://github.com/apache/arrow-datafusion/issues/2093#issuecomment-1080837823
@Ted-Jiang Another plausible or even preferable way is: we can branch in `in_list` to provide a fast-pass according to its args and reduce the burden it may take to maintain a separate function with a similar semantic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] yjshen commented on issue #2093: Add an `InSet` function as an optimized version for `IN`
Posted by GitBox <gi...@apache.org>.
yjshen commented on issue #2093:
URL: https://github.com/apache/arrow-datafusion/issues/2093#issuecomment-1081679093
FYI. Here's what Spark does: https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala#L579
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Ted-Jiang commented on issue #2093: Add an `InSet` function as an optimized version for `IN`
Posted by GitBox <gi...@apache.org>.
Ted-Jiang commented on issue #2093:
URL: https://github.com/apache/arrow-datafusion/issues/2093#issuecomment-1081625333
@yjshen I will take this 😊
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org