You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Hemanth Boyina (Jira)" <ji...@apache.org> on 2020/08/23 10:56:00 UTC
[jira] [Moved] (HADOOP-17219) ZStandardCodec compression mail
fail(generic error) when encounter specific file
[ https://issues.apache.org/jira/browse/HADOOP-17219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Boyina moved HDFS-15445 to HADOOP-17219:
------------------------------------------------
Component/s: (was: hdfs)
Key: HADOOP-17219 (was: HDFS-15445)
Affects Version/s: (was: 2.6.5)
2.6.5
Project: Hadoop Common (was: Hadoop HDFS)
> ZStandardCodec compression mail fail(generic error) when encounter specific file
> --------------------------------------------------------------------------------
>
> Key: HADOOP-17219
> URL: https://issues.apache.org/jira/browse/HADOOP-17219
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 2.6.5
> Environment: zstd 1.3.3
> hadoop 2.6.5
>
> --- a/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/zstd/TestZStandardCompressorDecompressor.java
> +++ b/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/zstd/TestZStandardCompressorDecompressor.java
> @@ -62,10 +62,8 @@
> @BeforeClass
> public static void beforeClass() throws Exception {
> CONFIGURATION.setInt(IO_FILE_BUFFER_SIZE_KEY, 1024 * 64);
> - uncompressedFile = new File(TestZStandardCompressorDecompressor.class
> - .getResource("/zstd/test_file.txt").toURI());
> - compressedFile = new File(TestZStandardCompressorDecompressor.class
> - .getResource("/zstd/test_file.txt.zst").toURI());
> + uncompressedFile = new File("/tmp/badcase.data");
> + compressedFile = new File("/tmp/badcase.data.zst");
> Reporter: Igloo
> Priority: Blocker
> Attachments: HDFS-15445.patch, badcase.data, image-2020-06-30-11-35-46-859.png, image-2020-06-30-11-39-17-861.png, image-2020-06-30-11-42-44-585.png, image-2020-06-30-11-51-18-026.png
>
>
> *Problem:*
> In our production environment, we put file in hdfs with zstd compressor, recently, we find that a specific file may leads to zstandard compressor failures.
> And we can reproduce the issue with specific file(attached file: badcase.data)
> !image-2020-06-30-11-51-18-026.png|width=1031,height=230!
>
> *Analysis*:
> ZStandarCompressor use buffersize( From zstd recommended compress out buffer size) for both inBufferSize and outBufferSize
> !image-2020-06-30-11-35-46-859.png|width=1027,height=387!
> but zstd indeed provides two separately recommending inputBufferSize and outputBufferSize
> !image-2020-06-30-11-39-17-861.png!
>
> *Workaround*
> One workaround, using recommended in/out buffer size provided by zstd lib can avoid the problem, but we don't know why.
> zstd recommended input buffer size: 1301072 (128 * 1024)
> zstd recommended ouput buffer size: 131591
> !image-2020-06-30-11-42-44-585.png|width=1023,height=196!
>
>
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org