You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@beam.apache.org by "Pei He (JIRA)" <ji...@apache.org> on 2016/11/03 19:56:58 UTC

[jira] [Created] (BEAM-894) Using @Teardown to remove temp files from failed bundles in Write.WriteBundles

Pei He created BEAM-894:
---------------------------

             Summary: Using @Teardown to remove temp files from failed bundles in Write.WriteBundles
                 Key: BEAM-894
                 URL: https://issues.apache.org/jira/browse/BEAM-894
             Project: Beam
          Issue Type: New Feature
          Components: sdk-java-core
            Reporter: Pei He
            Assignee: Pei He


FileBasedSink lefts temp files behind for failed bundles, and it forces finalize() to depend on pattern match.

However, pattern matching is not always reliable for eventual consistency file system, such as GCS.

Given we now have DoFn.TearDown, we can improve FileBasedSink (and in general Write transform) to remove temp files/resources early when DoFn bundles fail.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)