You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Michael Hogue (Jira)" <ji...@apache.org> on 2022/07/25 10:01:00 UTC
[jira] [Updated] (NIFI-10273) MergeContent tar compression fails on FlowFiles > 8.5GB
[ https://issues.apache.org/jira/browse/NIFI-10273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Hogue updated NIFI-10273:
---------------------------------
Description:
When attempting to tar compress FlowFiles > 8.5GB, MergeContent fails with the following error:
{noformat}
java.lang.RuntimeException: entry size '<FlowFileSize>' is too big (> 8589934591){noformat}
This is because apache commons-compress library has a hard maximum :
[https://commons.apache.org/proper/commons-compress/apidocs/src-html/org/apache/commons/compress/archivers/tar/TarConstants.html#line.94]
077777777777 octal bytes is 8589934591 decimal bytes (~8.5GB).
This can be addressed by configuring a BigNumberMode on the {{{}TarArchiveOutputStream{}}}:
[https://commons.apache.org/proper/commons-compress/javadocs/api-1.10/org/apache/commons/compress/archivers/tar/TarArchiveOutputStream.html#setBigNumberMode(int)]
}}
{{Here's an example of another project doing this:}}
{{[https://github.com/DataConservancy/dcs-packaging-tool/pull/36/files]}}
was:
When attempting to tar compress FlowFiles > 8.5GB, MergeContent fails with the following error:
{noformat}
java.lang.RuntimeException: entry size '<FlowFileSize>' is too big (> 8589934591){noformat}
This is because apache commons-compress library has a hard maximum :
[https://commons.apache.org/proper/commons-compress/apidocs/src-html/org/apache/commons/compress/archivers/tar/TarConstants.html#line.94]
077777777777 octal bytes is 8589934591 decimal bytes (~8.5GB).
This can be addressed by configuring a BigNumberMode on the {{TarArchiveOutputStream:
[https://commons.apache.org/proper/commons-compress/javadocs/api-1.10/org/apache/commons/compress/archivers/tar/TarArchiveOutputStream.html#setBigNumberMode(int)]
}}
{{Here's an example of another project doing this:}}
{{https://github.com/DataConservancy/dcs-packaging-tool/pull/36/files}}
> MergeContent tar compression fails on FlowFiles > 8.5GB
> -------------------------------------------------------
>
> Key: NIFI-10273
> URL: https://issues.apache.org/jira/browse/NIFI-10273
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Affects Versions: 1.16.3
> Reporter: Michael Hogue
> Priority: Minor
>
> When attempting to tar compress FlowFiles > 8.5GB, MergeContent fails with the following error:
> {noformat}
> java.lang.RuntimeException: entry size '<FlowFileSize>' is too big (> 8589934591){noformat}
> This is because apache commons-compress library has a hard maximum :
> [https://commons.apache.org/proper/commons-compress/apidocs/src-html/org/apache/commons/compress/archivers/tar/TarConstants.html#line.94]
> 077777777777 octal bytes is 8589934591 decimal bytes (~8.5GB).
> This can be addressed by configuring a BigNumberMode on the {{{}TarArchiveOutputStream{}}}:
> [https://commons.apache.org/proper/commons-compress/javadocs/api-1.10/org/apache/commons/compress/archivers/tar/TarArchiveOutputStream.html#setBigNumberMode(int)]
> }}
> {{Here's an example of another project doing this:}}
> {{[https://github.com/DataConservancy/dcs-packaging-tool/pull/36/files]}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)