You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "shenyu0127 (via GitHub)" <gi...@apache.org> on 2023/05/03 23:09:47 UTC
[GitHub] [pinot] shenyu0127 opened a new issue, #10718: Support transform functions in query preprocessing if all the arguments are all literals
shenyu0127 opened a new issue, #10718:
URL: https://github.com/apache/pinot/issues/10718
@npawar and I found this feature gap.
The Broker preprocess the query by invoking functions if all the function arguments are literals, and then passes the preprocessed query to the Server for query execution.
When we preprocess the query, we [only support scalar functions](https://github.com/apache/pinot/blob/d1227e466dc6e350214485334cdf4eedc7502d67/pinot-common/src/main/java/org/apache/pinot/sql/parsers/rewriter/CompileTimeFunctionsInvoker.java#L74). We should also support [transform functions](https://github.com/apache/pinot/tree/d1227e466dc6e350214485334cdf4eedc7502d67/pinot-core/src/main/java/org/apache/pinot/core/operator/transform/function).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] shenyu0127 commented on issue #10718: Support transform functions in query preprocessing if all the arguments are all literals
Posted by "shenyu0127 (via GitHub)" <gi...@apache.org>.
shenyu0127 commented on issue #10718:
URL: https://github.com/apache/pinot/issues/10718#issuecomment-1560292338
We didn't find any transform function missing the scalar function counterpart. We did not assume every transform function has a scalar function counterpart and assumed the feature gap.
If we can guarantee that every transform function has a scalar function counterpart, then:
- There is no feature gap.
- Why don't we make the transform functions depend on the scalar functions? The performance overhad should be minimal. The benefit is we ensure the two function won't diverge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] shenyu0127 commented on issue #10718: Support transform functions in query preprocessing if all the arguments are all literals
Posted by "shenyu0127 (via GitHub)" <gi...@apache.org>.
shenyu0127 commented on issue #10718:
URL: https://github.com/apache/pinot/issues/10718#issuecomment-1564749324
I am not proposing to use the combination of`ScalarTransformFunctionWrapper` and scalar functions to replace transform functions. I am proposing to make the transform functions call their corresponding scalar functions, e.g. make `DateTruncTransformFunction` call `DateTimeFunctions::datetrunc`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #10718: Support transform functions in query preprocessing if all the arguments are all literals
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #10718:
URL: https://github.com/apache/pinot/issues/10718#issuecomment-1564727056
> * Why don't we make the transform functions depend on the scalar functions? The performance overhead should be minimal. The benefit is we ensure the two function won't diverge.
Think of `TransformFunction` as a specialized implementation for better performance. There is a `ScalarTransformFunctionWrapper` which leverages the scalar functions so that if there is no specialized implementation for a scalar function, we fall back to using the scalar function. The performance overhead is not trivial because it always involves boxing the value, casting the type, and method invocation on a per record base.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #10718: Support transform functions in query preprocessing if all the arguments are all literals
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #10718:
URL: https://github.com/apache/pinot/issues/10718#issuecomment-1560220590
Transform function works on columnar data, and we should always have a scalar function counterpart for every transform function so that we can do row based transform.
Which transform function do you find that doesn't have the scalar function counterpart?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org