You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2020/05/19 20:42:48 UTC
[GitHub] [incubator-pinot] kishoreg opened a new issue #5416: Functions in Pinot
kishoreg opened a new issue #5416:
URL: https://github.com/apache/incubator-pinot/issues/5416
We have multiple types of Functions in Pinot
- AggregationFunction
- Used during query execution
- Stateful
- aggregate method can be invoked multiple times with blockValMap
- follows map/combine/reduce paradigm
- TransformFunction
- used during query execution
- stateless
- transform method is invoked with projectionBlock as input
- outputs another block (length of output block is same as input block)
- IngestionTransformFunction
- Used during ingestion (real-time and batch)
- stateless
- takes Generic Row and returns one value
- ScalarFunction (this is basically any java function)
- These are pure java functions that take a bunch of arguments and return a value
Most of the utility functions should be represented as ScalarFunctions
IngestTransformFunction, TransformFunction can be syntactic sugars on top of these scalar functions.
We are seeing the same function being needed in transform, ingestion, query parsing (e.g. date time functions) and we have to write the same version multiple times. Should we write generic wrappers around Scalar Function for ingestion, transform?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 edited a comment on issue #5416: Functions in Pinot
Posted by GitBox <gi...@apache.org>.
fx19880617 edited a comment on issue #5416:
URL: https://github.com/apache/incubator-pinot/issues/5416#issuecomment-631082032
My overall feeling is that we should just have two types of functions:
- aggregation: which knows how to do accumulate
- transform: which is pure transform, given value(s).
ingestion/scalar are just part of transform function with same/different apis/invoker.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mayankshriv commented on issue #5416: Functions in Pinot
Posted by GitBox <gi...@apache.org>.
mayankshriv commented on issue #5416:
URL: https://github.com/apache/incubator-pinot/issues/5416#issuecomment-631085745
We can create a hierarchy, with top level classification as:
1. Aggregation: m rows input k rows output (m <=k)
2. Transform: 1 row input 1 row output
Underneath 2 we can have, transform/scalar/ingestion
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on issue #5416: Functions in Pinot
Posted by GitBox <gi...@apache.org>.
fx19880617 commented on issue #5416:
URL: https://github.com/apache/incubator-pinot/issues/5416#issuecomment-631082032
My overall feeling is that we should just have two types of functions:
- aggregation: which knows how to do accumulate
- transform: which is pure transform, given value(s).
ingestion/scalar are just part of transform function with same/different apis.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 closed issue #5416: Functions in Pinot
Posted by GitBox <gi...@apache.org>.
fx19880617 closed issue #5416:
URL: https://github.com/apache/incubator-pinot/issues/5416
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org