You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "walterddr (via GitHub)" <gi...@apache.org> on 2023/11/21 16:19:45 UTC
[I] [bug][multistage] IN/NOT_IN function issue [pinot]
walterddr opened a new issue, #12035:
URL: https://github.com/apache/pinot/issues/12035
IN and NOT_IN operation has several caveats
1. it can be compiled into Sarg range search for example
- `floatField NOT IN (3, 5)` --> `Sarg{(-INF, 3) OR (3, 5) OR (5, +INF)}`
- `intField IN(1,2,3,4,10,11)` --> `Sarg{[1,4] OR [10,11]}`
2. it can be compiled without range search as non-range operation (just IN(field, literal_array) and NOT_IN(field, literal_array))
This posts several problems:
- not all range searches are efficient in Pinot
- NOT_IN operator is not implemented
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] [bug][multistage] IN/NOT_IN function issue [pinot]
Posted by "walterddr (via GitHub)" <gi...@apache.org>.
walterddr commented on issue #12035:
URL: https://github.com/apache/pinot/issues/12035#issuecomment-1863073495
one example
```
SELECT COUNT(*) FROM AirlineStats WHERE ARRAY_TO_MV(DivAirports) IN ('ONT', 'ACY', 'noodles', 'HNL') AND ARRAY_TO_MV(DivAirports) IN ('VLD') AND ARRAY_TO_MV(DivAirportSeqIDs) > 1432103
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] [bug][multistage] IN/NOT_IN function issue [pinot]
Posted by "walterddr (via GitHub)" <gi...@apache.org>.
walterddr closed issue #12035: [bug][multistage] IN/NOT_IN function issue
URL: https://github.com/apache/pinot/issues/12035
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] [bug][multistage] IN/NOT_IN function issue [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #12035:
URL: https://github.com/apache/pinot/issues/12035#issuecomment-1921768562
Find a failing query (`NOT IN` within `FILTER` clause):
```
SET useStarTree = false; SELECT OriginStateName, Dest, count(ArrivalDelayGroups) FILTER ( WHERE OriginStateName NOT IN ('Michigan', 'Utah', 'Montana', 'Kentucky')), avg(Cancelled), sum(WeatherDelay) FROM mytable WHERE Flights BETWEEN 1 AND 1 AND Flights <> 1 AND OriginCityName >= 'Jackson/Vicksburg, MS' AND OriginStateName <= 'South Dakota' GROUP BY OriginStateName, Dest ORDER BY OriginStateName, Dest
```
Exception:
```
java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Cannot find function with name: NOT_IN
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2096)
at org.apache.pinot.query.service.server.QueryServer.submit(QueryServer.java:125)
at org.apache.pinot.common.proto.PinotQueryWorkerGrpc$MethodHandlers.invoke(PinotQueryWorkerGrpc.java:284)
...
Caused by: java.lang.IllegalStateException: Cannot find function with name: NOT_IN
at com.google.common.base.Preconditions.checkState(Preconditions.java:601)
at org.apache.pinot.query.runtime.operator.operands.FunctionOperand.<init>(FunctionOperand.java:51)
at org.apache.pinot.query.runtime.operator.operands.TransformOperandFactory.getTransformOperand(TransformOperandFactory.java:77)
at org.apache.pinot.query.runtime.operator.operands.TransformOperandFactory.getTransformOperand(TransformOperandFactory.java:34)
org.apache.pinot.query.service.dispatch.QueryDispatcher.submit(QueryDispatcher.java:148)
org.apache.pinot.query.service.dispatch.QueryDispatcher.submitAndReduce(QueryDispatcher.java:92)
org.apache.pinot.broker.requesthandler.MultiStageBrokerRequestHandler.handleRequest(MultiStageBrokerRequestHandler.java:191)
org.apache.pinot.broker.requesthandler.BaseBrokerRequestHandler.handleRequest(BaseBrokerRequestHandler.java:288)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org