You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2020/11/04 23:55:37 UTC
[GitHub] [incubator-pinot] Jackie-Jiang opened a new issue #6240: Epoch time format does not work with size other than 1
Jackie-Jiang opened a new issue #6240:
URL: https://github.com/apache/incubator-pinot/issues/6240
Epoch time format has the following components: `[size]:[unit]:EPOCH`
E.g. `1:DAYS:EPOCH`, `5:MINUTES:EPOCH`
This size info is not preserved In the `SegmentGeneratorConfig` and `SegmentMetadata`, thus the generated segment won't have the correct time metadata when the size is not 1.
We have never seen a use case with size other than 1, so we might just want to drop the size info from the time format. Also, the unit info does not apply to the simple-date-format, so I would recommend changing the time format to:
`EPOCH:[unit]` for epoch time and `SIMPLE_DATE_FORMAT:[format]` for simple-date-format.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] npawar commented on issue #6240: Epoch time format does not work with size other than 1
Posted by GitBox <gi...@apache.org>.
npawar commented on issue #6240:
URL: https://github.com/apache/incubator-pinot/issues/6240#issuecomment-728555875
Suggested change looks good to me. I have 2 concerns
1. The dateTimeConvert udf is using this convention as well, and I believe is directly using the DateTimeFieldSpec and related classes. We might have to make changes to maintain that udf.
2. I see some usages of getColumnUnit, without checking the format first. There will be some thought needed to decide what happens to those.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] BACtaki commented on issue #6240: Epoch time format does not work with size other than 1
Posted by GitBox <gi...@apache.org>.
BACtaki commented on issue #6240:
URL: https://github.com/apache/incubator-pinot/issues/6240#issuecomment-727084306
@Jackie-Jiang is anybody looking into it already? If not, then I'd like to work on this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on issue #6240: Epoch time format does not work with size other than 1
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #6240:
URL: https://github.com/apache/incubator-pinot/issues/6240#issuecomment-727115326
@npawar What's your thought on changing the time format to the one I proposed? We have a volunteer
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mcvsubbu commented on issue #6240: Epoch time format does not work with size other than 1
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on issue #6240:
URL: https://github.com/apache/incubator-pinot/issues/6240#issuecomment-722723112
OK, my bad. We don't have a use case like this.
But, for those who do, what is your migration path?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on issue #6240: Epoch time format does not work with size other than 1
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #6240:
URL: https://github.com/apache/incubator-pinot/issues/6240#issuecomment-722563255
@mcvsubbu Then we can use that to verify the behavior. Please check the segment metadata / ZK metadata and see if the start/end time is converted to epoch in minutes in order for it to behave correctly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] mcvsubbu commented on issue #6240: Epoch time format does not work with size other than 1
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on issue #6240:
URL: https://github.com/apache/incubator-pinot/issues/6240#issuecomment-722073263
I am pretty sure we have a use case with 5:minutes:epoch, I need to confirm. It uses this to transform the time column upon ingestion
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on issue #6240: Epoch time format does not work with size other than 1
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on issue #6240:
URL: https://github.com/apache/incubator-pinot/issues/6240#issuecomment-726454231
This format won't be correctly handled, and the segment will be deleted by the retention manager mistakenly. So I don't think we need a migration path for it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org