[GitHub] [nifi] exceptionfactory commented on pull request #6369: NIFI-10273: added support for files larger than 8.5GB to MergeContent…

[GitBox <gi...@apache.org>]

exceptionfactory commented on PR #6369:
URL: https://github.com/apache/nifi/pull/6369#issuecomment-1241211675

   Thanks for the background and runtime testing on this issue @michael81877.
   
   The change seem straightforward, and I agree that attempting to build a unit test for file that large is not helpful.
   
   Reviewing the [Commons Compress TarArchiveOutputStream](https://commons.apache.org/proper/commons-compress/apidocs/org/apache/commons/compress/archivers/tar/TarArchiveOutputStream.html#BIGNUMBER_POSIX) documentation and source code, setting the `BIGNUMBER_POSIX` mode unconditionally writes the TAR entry headers using a [different approach](https://pubs.opengroup.org/onlinepubs/009695399/utilities/pax.html#tag_04_100_13_03).
   
   According to references, it seems like this format has been around long enough that it is supported in modern TAR utilities. With that being said, there is the outside possibility that this change could cause complications with some legacy system.
   
   The `TarMerge.merge()` method has a reference to all the FlowFiles that will be written, so it seems possible to calculate the total size prior to setting the mode. By calculating the total and checking it against a value, it would be possible to set this flag only when the output is expected to exceed the maximum size. What do you think of that approach?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org