You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/06/08 18:24:01 UTC
[GitHub] [pinot] kkrugler opened a new issue, #8864: Support cleaning up segments when a batch build fails
kkrugler opened a new issue, #8864:
URL: https://github.com/apache/pinot/issues/8864
Currently if segment generation fails for one of the input files, segments created from previous input files will be left in the output directory. A new `cleanupOnFailure` flag (default is false) would trigger the deletion of these segments. Note that clearing the output directory isn't acceptable, as there might be existing files.
This change would have to be applied to all batch segment generation plugins (stand-alone, Hadoop, Spark).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] stym06 commented on issue #8864: Support cleaning up segments when a batch build fails
Posted by GitBox <gi...@apache.org>.
stym06 commented on issue #8864:
URL: https://github.com/apache/pinot/issues/8864#issuecomment-1154206654
Let me know if I can take this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] kkrugler commented on issue #8864: Support cleaning up segments when a batch build fails
Posted by GitBox <gi...@apache.org>.
kkrugler commented on issue #8864:
URL: https://github.com/apache/pinot/issues/8864#issuecomment-1154555649
@stym06 - sure, go for it! One challenge is that testing this is painful, as you need to manually run stand-alone, Hadoop and Spark jobs to really validate proper behavior. Sadly, I don't think we've got integration tests for Hadoop or Spark, though I think I saw a recent issue about addressing that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org