You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@beam.apache.org by "Luke Cwik (JIRA)" <ji...@apache.org> on 2017/05/02 14:19:04 UTC

[jira] [Created] (BEAM-2138) Support atomic rename within FileSystem to replace inefficient Hadoop copy

Luke Cwik created BEAM-2138:
-------------------------------

             Summary: Support atomic rename within FileSystem to replace inefficient Hadoop copy
                 Key: BEAM-2138
                 URL: https://issues.apache.org/jira/browse/BEAM-2138
             Project: Beam
          Issue Type: Improvement
          Components: sdk-java-core, sdk-java-extensions
            Reporter: Luke Cwik


Hadoop copy operation is inefficient since it needs to stream the entirety of the resource through the machine performing the copy. Hadoop file system implementations do support an efficient rename.

Apache Beam sinks rely on being able to rename files atomically which is currently done by using FileSystem copy + delete.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)