You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/05/26 01:15:09 UTC
[GitHub] [incubator-pinot] jtao15 opened a new issue #6979: Time partition support for SegmentProcessorFramework
jtao15 opened a new issue #6979:
URL: https://github.com/apache/incubator-pinot/issues/6979
Result segments need to be time aligned for some minion tasks (e.g. backfill support for merge/rollup), it will be good if we can add time partition support based on the time window/bucket.
The configs can be something like below:
1. window start time (in milliseconds)
2. time bucket/list of time buckets (in milliseconds)
Say the input n segments have data between 02/2021 to 05/2021, and we want to process all the segments to be monthly aligned, we can set the config as:
```
{
startTime: 02/01/2021 00:00:00,
timeBuckets: [28d, 31d, 30d, 31d]
}
```
Each result segment will only contain data in a certain month.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] snleee commented on issue #6979: Time partition support for SegmentProcessorFramework
Posted by GitBox <gi...@apache.org>.
snleee commented on issue #6979:
URL: https://github.com/apache/incubator-pinot/issues/6979#issuecomment-848387224
1. `SegmentProcessorFramework` does not need to know about the time buckets or start time. The only input config that we need is `timeGranularity (e.g. daily, monthly...etc)`.
2. Internally, the segment processor framework should correctly transform time column value and partition it accordingly based on the output partition value.
```
e.g. input data between 02/2021 to 05/2021 , use monthly time bucket
time_column_value | partition value
05/21/2021 12:13:12 | 05/01/2021
04/13/2021 14:34:6 | 04/01/2021
...
```
If we make sure that all the data with the same partition value goes to the same file, then we are good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] jtao15 edited a comment on issue #6979: SegmentProcessorFramework improvements for merge/rollup
Posted by GitBox <gi...@apache.org>.
jtao15 edited a comment on issue #6979:
URL: https://github.com/apache/incubator-pinot/issues/6979#issuecomment-849869980
Besides the time partition, it's good to make the resulted segment name configurable also. Listing down the requirements of the improvements:
1. Time partition
- Partition bucket granularity (Hourly, Daily, Weekly, Monthly, Yearly) should be configurable.
- Each mapper directory can only contains data in one time bucket. (E.g. merge segments (from 02/2021 to 05/2021) by month, there should be 4 mapper phase result directories: 02/2021, 03/2021, 04/2021, 05/2021)
2. Segment name
- The name patterns (prefixes and timestamp/uuid) of result segments should be configurable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] jtao15 commented on issue #6979: SegmentProcessorFramework improvements for merge/rollup
Posted by GitBox <gi...@apache.org>.
jtao15 commented on issue #6979:
URL: https://github.com/apache/incubator-pinot/issues/6979#issuecomment-849869980
Besides the time partition, it's good to make the resulted segment name configurable also. Listing down the requirements of the improvements:
1. Time partition
- Partition bucket granularity (Hourly, Daily, Weekly, Monthly, Yearly) should be configurable.
- Each mapper directory can only contains data in one time bucket. (E.g. merge segments (from 02/2021 to 05/2021) by month, there should be 4 mapper phase result directories: 02/2021, 03/2021, 04/2021, 05/2021)
2. Segment name
- The name patterns or prefixes of result segments should be configurable.
- The result segments of minion tasks should not collide with current segments.
- The segments between minion tasks (same or different task types) should not collide with each other.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org