You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/09/09 21:03:07 UTC
[GitHub] [pinot] jugomezv opened a new issue, #9366: [Feature] Support time-based backfills
jugomezv opened a new issue, #9366:
URL: https://github.com/apache/pinot/issues/9366
The purpose of this work is to support time-based backfill operations for append-tables. Currently backfills are based on segment name matching: once this feature is added we will be able to backfill periods of time in tables, regardless of whether the segment names match or not.
**Requirements:**
1. Provide support for time-based backfill operations for offline tables and completed segments for real-time tables.
2. This feature should be compatible with Consistent Push and Segment Merge and Rollup.
3. Check that segments uploaded are within the time interval specified.
**Not Supported by this work:**
1. Time-based backfills of data for intervals that intersect segment boundaries are not supported: that is all segments to be replaced must be fully contained in the time interval provided.
2. Backfill of data that span the time interval of consuming segments. What this means is we do not support backfill operations that overlap real time data that has not been turned into completed segments.
3. Merge and rollup operations started during a backfill that could rewind the watermark used for merge and rollups.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] jugomezv commented on issue #9366: [Feature] Support time-based backfills
Posted by GitBox <gi...@apache.org>.
jugomezv commented on issue #9366:
URL: https://github.com/apache/pinot/issues/9366#issuecomment-1245686155
First step toward this feature is to create a function that can list the all the segments fully contained in an interval of time provided.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org