You are viewing a plain text version of this content. The canonical link for it is here.

Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/03 17:18:13 UTC

[GitHub] [beam] kennknowles opened a new issue, #18280: Support atomic rename within FileSystem to replace inefficient Hadoop copy

kennknowles opened a new issue, #18280:
URL: https://github.com/apache/beam/issues/18280

   Hadoop copy operation is inefficient since it needs to stream the entirety of the resource through the machine performing the copy. Hadoop file system implementations do support an efficient rename.
   
   Apache Beam sinks rely on being able to rename files atomically which is currently done by using FileSystem copy **** delete.
   
   Imported from Jira [BEAM-2138](https://issues.apache.org/jira/browse/BEAM-2138). Original Jira may contain additional context.
   Reported by: lcwik.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org