[GitHub] [pinot] jackjlli commented on pull request #8240: allow skip proactive cleaning segment lineages

[GitBox <gi...@apache.org>]

jackjlli commented on pull request #8240:
URL: https://github.com/apache/pinot/pull/8240#issuecomment-1048333326


   I think what you're trying to do here is to support consistent push for append use cases, e.g. for daily data push. If that's the case, the logic should handle the following scenarios:
   * For refresh table, it's always not allowed to have two concurrent push happening at the same time, regardless of the emptiness of the existing table. In this case, the previous `IN_PROGRESS` segment lineage should be cleaned up.
   * For append table, it's okay to have more than one batch push at the same time, as long as they're not pushing for the same set of data (i.e. they're not overwriting each other).
   
   So going back to the code, I think it'd be good to include the validation above instead of just releasing the `forceCleanup` to false. cc: @snleee @jtao15 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org