You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/11/16 23:56:46 UTC
[GitHub] [pinot] amrishlal opened a new issue, #9820: Numerical type conversion of literal values in IN and NOT_IN clause
amrishlal opened a new issue, #9820:
URL: https://github.com/apache/pinot/issues/9820
Pinot currently supports numerical type conversion of literal values in WHERE clause as shown in the examples below:
```
select runs from baseballStats WHERE runs = 10.0
select runs from baseballStats WHERE runs = 10.3
select runs from baseballStats WHERE runs != 10.3
select runs from baseballStats WHERE runs > 6.5 AND runs < 10.23
```
This support was added through PR https://github.com/apache/pinot/pull/6811 and https://github.com/apache/pinot/pull/6927. However, Pinot still doesn't support type conversion of literal values that may appear in the IN clause. For example, the queries shown below will throw exceptions:
```
select runs from baseballStats WHERE runs IN (5.0, 6, 7.2)
select runs from baseballStats WHERE runs NOT IN (5.0, 6, 7.2)
```
We would like to extend the type conversion mechanism in `NumericalFilterOptimizer` to cover IN and NOT_IN clauses as well. One side effect of this change would be that we would need to make another pass over all the literal values specified in IN or NOT_IN clause when these clauses are present in the query.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] amrishlal commented on issue #9820: Numerical type conversion of literal values in IN and NOT_IN clause
Posted by GitBox <gi...@apache.org>.
amrishlal commented on issue #9820:
URL: https://github.com/apache/pinot/issues/9820#issuecomment-1324397213
> After a second thought, should we consider handling it in the PredicateEvaluatorFactory where the data type is already known? That way we can also handle arbitrary expressions on the left-hand side.
If LHS is an expression, then things seem to work as in the statement `select runs from baseballStats where 2*runs IN (310.0, 300.0)`. I think this works because, the LHS is of type DOUBLE and hence `BaseRawValueBasedPredicateEvaluator.newRawValueBasedEvaluator` converts all IN clause values into DOUBLE and converting any numerical value to DOUBLE is a valid conversion in Pinot.
However, `select runs from baseballStats where runs IN (310.0, 300.0)` fails in `ColumnValueSegmentPruner.pruneInPredicate` because LHS is an INT column and we can't convert 310.0 and 300.0 to INT.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #9820: Numerical type conversion of literal values in IN and NOT_IN clause
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #9820:
URL: https://github.com/apache/pinot/issues/9820#issuecomment-1322885726
After a second thought, should we consider handling it in the `PredicateEvaluatorFactory` where the data type is already known? That way we can also handle arbitrary expressions on the left-hand side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #9820: Numerical type conversion of literal values in IN and NOT_IN clause
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #9820:
URL: https://github.com/apache/pinot/issues/9820#issuecomment-1324399426
Not all expressions work though. If the result of the expression is type `INT`, then it won't work, and the current optimizer cannot fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] amrishlal commented on issue #9820: Numerical type conversion of literal values in IN and NOT_IN clause
Posted by GitBox <gi...@apache.org>.
amrishlal commented on issue #9820:
URL: https://github.com/apache/pinot/issues/9820#issuecomment-1319105276
FYI: @siddharthteotia @Jackie-Jiang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] amrishlal commented on issue #9820: Numerical type conversion of literal values in IN and NOT_IN clause
Posted by GitBox <gi...@apache.org>.
amrishlal commented on issue #9820:
URL: https://github.com/apache/pinot/issues/9820#issuecomment-1330067945
Discussed briefly with @Jackie-Jiang offline. Jackie mentioned that these proposed changes in `NumericalFilterOptimizer` won't apply to cases where the LHS of the predicate is not DOUBLE (and this can happen currently when functions like `timeConvert` and `arrayLength` are used on LHS). The current plan is to make these changes in `NumericalFilterOptimizer` as interim fixes and then follow up with changes in `PredicateEvaluatorFactory` and `ColumnValueSegmentPruner` that would ultimately allow us to take care of type determination with LHS expressions as well. @Jackie-Jiang Let me know if I am missing anything. cc: @siddharthteotia
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org