You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@nifi.apache.org by "Simon Bence (Jira)" <ji...@apache.org> on 2021/09/29 15:35:00 UTC

[jira] [Created] (NIFI-9260) Making "write and rename" behaviour optional in PutHDFS

Simon Bence created NIFI-9260:
---------------------------------

             Summary: Making "write and rename" behaviour optional in PutHDFS
                 Key: NIFI-9260
                 URL: https://issues.apache.org/jira/browse/NIFI-9260
             Project: Apache NiFi
          Issue Type: Improvement
          Components: Extensions
            Reporter: Simon Bence
            Assignee: Simon Bence


In some cases the current "write and rename" behaviour of PutHDFS undesirable. Particularly if the the server is backed with AWS S3 which has no "native" rename but copies file content when renaming, serious perfomance issues might arise.

In order to avoid this, I propose to make this behaviour optional and make it possible for the PutHDFS to write directly to the final destination. Of course, depending on the use case, there is the risk of reading "half-written" files in this manner, but in some scenarios this is not a danger. Also, in case of S3 this should not be an issue as files in S3 are visible only after the "write" operation is finished



--
This message was sent by Atlassian Jira
(v8.3.4#803005)