[jira] [Commented] (BEAM-894) Using @Teardown to remove temp files from failed bundles in Write.WriteBundles

["Pei He (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15634041#comment-15634041 ] 

Pei He commented on BEAM-894:
-----------------------------

This will also help the redesign of IOChannelFactory: https://issues.apache.org/jira/browse/BEAM-59

> Using @Teardown to remove temp files from failed bundles in Write.WriteBundles
> ------------------------------------------------------------------------------
>
>                 Key: BEAM-894
>                 URL: https://issues.apache.org/jira/browse/BEAM-894
>             Project: Beam
>          Issue Type: New Feature
>          Components: sdk-java-core
>            Reporter: Pei He
>            Assignee: Pei He
>
> FileBasedSink lefts temp files behind for failed bundles, and it forces finalize() to depend on pattern match.
> However, pattern matching is not always reliable for eventual consistency file system, such as GCS.
> Given we now have DoFn.TearDown, we can improve FileBasedSink (and in general Write transform) to remove temp files/resources early when DoFn bundles fail.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)