You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/03/22 21:06:17 UTC
[GitHub] [incubator-pinot] mqliang opened a new issue #6708: Support ago() transform function
mqliang opened a new issue #6708:
URL: https://github.com/apache/incubator-pinot/issues/6708
Pinot supports two types of transform functions:
* Scalar transforms
* Non-scalar transforms
Scalar transform take in a scalar (string, int etc) value and output a scalar value. For example, now() was added as a scalar transform function to return the current timestamp as milliseconds from epoch
We should consider adding another scalar function ago(timespan) which subtracts the given timespan time from the current timestamp
E.g
* ago(1h) should subtract 1 hour from current time
* ago(1d) should subtract 1 day from current time
We can then potentially use it in filter clauses as
WHERE timeCol > ago(1h) -> this will return all rows with time in the last 1 hour.
The current alternative to do this is WHERE timeCol > toEpochHours(now()) - 1
But ago() can also be a good addition to our list of inbuilt scalar functions
Considerations
* See how the interval passed to ago() as argument should be specified.
* What should be the return value of ago() ? Always millisSinceEpoch or the value corresponding to the unit passed in the argument ? For example, if the user passes ago(1h) then it subtracts 1 hour from the current time and returns the value as hoursSinceEpoch. Similarly, if the user does ago(1d), then it subtracts 1 day from the current time and returns the value as daysSinceEpoch.
* What happens if the timeCol we are comparing with is not in the same units ? For example, if user is doing WHERE timeCol > ago(1h) then it is expected that timeCol has values in hoursSinceEpoch. Looks like this and previous point are somewhat related
cc @siddharthteotia
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on issue #6708: Support ago() transform function
Posted by GitBox <gi...@apache.org>.
fx19880617 commented on issue #6708:
URL: https://github.com/apache/incubator-pinot/issues/6708#issuecomment-804416939
I think @Jackie-Jiang is on adding a new data type for DATE/TIMESTAMP.
From my perspective, the reason we have so many time unit conversion functions is that we push all the overhead of managing type conversion to users.
Ideally, those functions should either return type of TIMESTAMP or DATE, then we can handle the comparison internally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia closed issue #6708: Support ago() transform function
Posted by GitBox <gi...@apache.org>.
siddharthteotia closed issue #6708:
URL: https://github.com/apache/incubator-pinot/issues/6708
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] fx19880617 commented on issue #6708: Support ago() transform function
Posted by GitBox <gi...@apache.org>.
fx19880617 commented on issue #6708:
URL: https://github.com/apache/incubator-pinot/issues/6708#issuecomment-804429049
I prefer to make `ago` returns the value in a fixed time unit like millis so it will be simpler for users to modify it and later on we may easily do directory convert over it. E.g. we can convert millis value directory to `TIMESTAMP` type in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on issue #6708: Support ago() transform function
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on issue #6708:
URL: https://github.com/apache/incubator-pinot/issues/6708#issuecomment-804427956
So for now, may be we can settle with semantic as follows:
For a filter `WHERE timeCol > ago(1h)`, we will convert current time (millis from epoch) into hours and then do subtraction. We will use assume that timeCol has values in hoursSinceEpoch or if not user then user does the conversion `WHERE toEpochHours(timeCol) > ago(1h)`. This can be applied to other timespan interval arguments (1d, 1h, 1m, 1s) to ago()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on issue #6708: Support ago() transform function
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on issue #6708:
URL: https://github.com/apache/incubator-pinot/issues/6708#issuecomment-881704413
Support added in https://github.com/apache/incubator-pinot/pull/6820
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org