[GitHub] [pinot] walterddr commented on pull request #8176: add retention period to deleted segment files and allow table level o…

[GitBox <gi...@apache.org>]

walterddr commented on pull request #8176:
URL: https://github.com/apache/pinot/pull/8176#issuecomment-1034030564


   > This PR is complicating things and trying to solve a problem that may never exist.
   
   yes this requirement does exist. what we want to control is the "undo" period for a table after deletion.
   
   > It is fair to NOT expect that deletion of a table behave as per table config settings. it is ok to use the cluster behavior if the table config is not found. 
   
   Sorry let me re-clarify: the issue is that when the deletion retention period changes (whether cluster level or table level), does that retroactively apply to segments that has already been deleted. think of this sequence:
   1. a segment is deleted at timestamp `now() - 2d`, at the time the retention setting was `7d`
   2. cluster retention config changed to `1d` at timestamp `now() - 1d`. 
   
   now question: does this apply to the segment that was deleted at `now() - 2d` ?  if so, then the segment will be immediately deleted, otherwise we should still honor the retention setting `7d` at the time this segment was deleted. 
   
   > If it is desired that the segments be re-loaded onto another table, the operator should save the segments before creating a new table.
   
   reloading on to another table is not the problem I was trying to address.
   
   > 
   > Also, I don't understand the need to re-scan all segments when table config retention time is changed.
   
   this was echo back to the example i said above: if we decided to retroactively apply the new retention period to previously deleted segments, we then need to read out all the DELETED_SEGMENTS, find their table-level override and the cluster level config. apply the retention deletion rule if they had already gone pass the new retention time. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org