[jira] [Created] (NIFI-4775) Allow FlowFile Repository to optionally perform fsync when writing CREATE events but not other events

["Mark Payne (JIRA)" <ji...@apache.org>]

Mark Payne created NIFI-4775:
--------------------------------

             Summary: Allow FlowFile Repository to optionally perform fsync when writing CREATE events but not other events
                 Key: NIFI-4775
                 URL: https://issues.apache.org/jira/browse/NIFI-4775
             Project: Apache NiFi
          Issue Type: Improvement
          Components: Core Framework
            Reporter: Mark Payne


Currently, when a FlowFile is written to the FlowFile Repository, the repo can either fsync or not, depending on nifi.properties. We should allow a third option, of fsync only for CREATE events. In this case, if we receive new data from a source we can fsync the update to the FlowFile Repository before ACK'ing the data from the source. This allows us to guarantee data persistence without the overhead of an fsync for every FlowFile Repository update.

It may make sense, though, to be a bit more selective about when do this. For example if the source is a system that does not allow us to acknowledge the receipt of data, such as a ListenUDP processor, this doesn't really buy us much. In such a case, we could be smart about avoiding the high cost of an fsync. However, for something like GetSFTP where we have to remove the file in order to 'acknowledge receipt' we can ensure that we wait for the fsync before proceeding.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)