[GitHub] [nifi] markap14 commented on a change in pull request #4792: NIFI-8183 TailFile intermittently creates records with ascii NULL after rollover

[GitBox <gi...@apache.org>]

markap14 commented on a change in pull request #4792:
URL: https://github.com/apache/nifi/pull/4792#discussion_r569490344



##########
File path: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/TailFile.java
##########
@@ -1184,7 +1191,15 @@ private boolean recoverRolledFiles(final ProcessContext context, final ProcessSe
                             // This is the same file that we were reading when we shutdown. Start reading from this point on.
                             rolledOffFiles.remove(0);
                             FlowFile flowFile = session.create();
-                            flowFile = session.importFrom(in, flowFile);
+                            try {
+                                ByteArrayOutputStream out = new ByteArrayOutputStream();

Review comment:
       This approach is going to buffer everything that gets read into the ByteArrayOutputStream. This could potentially be a huge amount of data. We want to be sure that we don't buffer that up. Instead, I would recommend an approach like:
   ```
   try (final OutputStream out = session.write(flowFile)) {
     readLines(fis.getChannel(), ByteBuffer.allocate(65536), out, new CRC32(), reReadOnNul, true);
   }
   ```
   This allows us to write directly to the content repository instead of buffering the data in a ByteArrayOutputStream.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org