You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Kenneth Knowles (Jira)" <ji...@apache.org> on 2022/06/03 19:23:00 UTC

[jira] [Commented] (BEAM-3145) Improve cleanup of zombie temporary files in WriteFiles

    [ https://issues.apache.org/jira/browse/BEAM-3145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17546743#comment-17546743 ] 

Kenneth Knowles commented on BEAM-3145:
---------------------------------------

This issue has been migrated to https://github.com/apache/beam/issues/18577

> Improve cleanup of zombie temporary files in WriteFiles
> -------------------------------------------------------
>
>                 Key: BEAM-3145
>                 URL: https://issues.apache.org/jira/browse/BEAM-3145
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-core
>            Reporter: Eugene Kirpichov
>            Priority: P3
>              Labels: Clarified
>
> See user issue in https://stackoverflow.com/questions/47113773/dataflow-2-1-0-streaming-application-is-not-cleaning-temp-folders
> For windowed writes, the proper solution is probably to put temp files into finer-grained directories, e.g. sharded by date or hour, and somehow clean them up by globbing and deleting the entire directory when the watermark goes past said date. It's complicated by late data and multiple trigger firings, of course.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)