[jira] [Created] (NIFI-32) Swap File format should include summary at end for faster NiFi system startup

["Mark Payne (JIRA)" <ji...@apache.org>]

Mark Payne created NIFI-32:
------------------------------

             Summary: Swap File format should include summary at end for faster NiFi system startup
                 Key: NIFI-32
                 URL: https://issues.apache.org/jira/browse/NIFI-32
             Project: Apache NiFi
          Issue Type: Improvement
            Reporter: Mark Payne
            Priority: Minor


When NiFi is restarted with many hundreds or thousands of Swap Files, it can take a very long time to startup.

If when we write out the Swap Files, we append a summary of the Swap File (queue identifier, number of FlowFiles, size of FlowFiles, and ContentClaim info/count map), we can then read far less data for restart. This means we would serialize the summary to a ByteArrayOutputStream, then get the length, and then write out the summary, followed by length. On restore, we read just the last 8 bytes, and that allows us to generate the offset into the file to start reading the summary.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)