You are viewing a plain text version of this content. The canonical link for it is here.

Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/03 19:22:53 UTC

[GitHub] [beam] kennknowles opened a new issue, #18577: Improve cleanup of zombie temporary files in WriteFiles

kennknowles opened a new issue, #18577:
URL: https://github.com/apache/beam/issues/18577

   See user issue in https://stackoverflow.com/questions/47113773/dataflow-2-1-0-streaming-application-is-not-cleaning-temp-folders
   
   For windowed writes, the proper solution is probably to put temp files into finer-grained directories, e.g. sharded by date or hour, and somehow clean them up by globbing and deleting the entire directory when the watermark goes past said date. It's complicated by late data and multiple trigger firings, of course.
   
   Imported from Jira [BEAM-3145](https://issues.apache.org/jira/browse/BEAM-3145). Original Jira may contain additional context.
   Reported by: jkff.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org