You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Pankaj Kumar (Jira)" <ji...@apache.org> on 2024/07/23 14:23:00 UTC
[jira] [Updated] (HBASE-28655) TestHFileCompressionZstd fails with IllegalArgumentException: Illegal bufferSize
[ https://issues.apache.org/jira/browse/HBASE-28655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pankaj Kumar updated HBASE-28655:
---------------------------------
Description:
HADOOP-18810 added io.compression.codec.zstd.buffersize in core-default.xml with default value as 0.
So ZSTD buffer size will be returned as 0 based on core-default.xml,
{code:java}
static int getBufferSize(Configuration conf) {
return conf.getInt(ZSTD_BUFFER_SIZE_KEY,
conf.getInt(CommonConfigurationKeys.IO_COMPRESSION_CODEC_ZSTD_BUFFER_SIZE_KEY,
// IO_COMPRESSION_CODEC_ZSTD_BUFFER_SIZE_DEFAULT is 0! We can't allow that.
ZSTD_BUFFER_SIZE_DEFAULT));
}
{code}
HBASE-26259 added a value check, but got reverted in HBASE-26959.
This issue will also occur during region flush and abort the RegionServer.
TestHFileCompressionZstd and other zstd related test cases are are also failing,
{code:java}
java.lang.IllegalArgumentException: Illegal bufferSize
at org.apache.hadoop.io.compress.CompressorStream.<init>(CompressorStream.java:42)
at org.apache.hadoop.io.compress.BlockCompressorStream.<init>(BlockCompressorStream.java:56)
at org.apache.hadoop.hbase.io.compress.aircompressor.ZstdCodec.createOutputStream(ZstdCodec.java:106)
at org.apache.hadoop.hbase.io.compress.Compression$Algorithm.createPlainCompressionStream(Compression.java:454)
at org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultEncodingContext.<init>(HFileBlockDefaultEncodingContext.java:99)
at org.apache.hadoop.hbase.io.hfile.NoOpDataBlockEncoder.newDataBlockEncodingContext(NoOpDataBlockEncoder.java:85)
at org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.<init>(HFileBlock.java:846)
at org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.finishInit(HFileWriterImpl.java:304)
at org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.<init>(HFileWriterImpl.java:185)
at org.apache.hadoop.hbase.io.hfile.HFile$WriterFactory.create(HFile.java:312)
at org.apache.hadoop.hbase.io.compress.HFileTestBase.doTest(HFileTestBase.java:73)
at org.apache.hadoop.hbase.io.compress.aircompressor.TestHFileCompressionZstd.test(TestHFileCompressionZstd.java:54)
{code}
was:
HADOOP-18810 added io.compression.codec.zstd.buffersize in core-default.xml with default value as 0.
So ZSTD buffer size will be returned as 0 based on core-default.xml,
{code}
static int getBufferSize(Configuration conf) {
return conf.getInt(ZSTD_BUFFER_SIZE_KEY,
conf.getInt(CommonConfigurationKeys.IO_COMPRESSION_CODEC_ZSTD_BUFFER_SIZE_KEY,
// IO_COMPRESSION_CODEC_ZSTD_BUFFER_SIZE_DEFAULT is 0! We can't allow that.
ZSTD_BUFFER_SIZE_DEFAULT));
}
{code}
HBASE-26259 added a value check, but got reverted in HBASE-26959.
TestHFileCompressionZstd and other zstd related test cases are failing,
{code}
java.lang.IllegalArgumentException: Illegal bufferSize
at org.apache.hadoop.io.compress.CompressorStream.<init>(CompressorStream.java:42)
at org.apache.hadoop.io.compress.BlockCompressorStream.<init>(BlockCompressorStream.java:56)
at org.apache.hadoop.hbase.io.compress.aircompressor.ZstdCodec.createOutputStream(ZstdCodec.java:106)
at org.apache.hadoop.hbase.io.compress.Compression$Algorithm.createPlainCompressionStream(Compression.java:454)
at org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultEncodingContext.<init>(HFileBlockDefaultEncodingContext.java:99)
at org.apache.hadoop.hbase.io.hfile.NoOpDataBlockEncoder.newDataBlockEncodingContext(NoOpDataBlockEncoder.java:85)
at org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.<init>(HFileBlock.java:846)
at org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.finishInit(HFileWriterImpl.java:304)
at org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.<init>(HFileWriterImpl.java:185)
at org.apache.hadoop.hbase.io.hfile.HFile$WriterFactory.create(HFile.java:312)
at org.apache.hadoop.hbase.io.compress.HFileTestBase.doTest(HFileTestBase.java:73)
at org.apache.hadoop.hbase.io.compress.aircompressor.TestHFileCompressionZstd.test(TestHFileCompressionZstd.java:54)
{code}
> TestHFileCompressionZstd fails with IllegalArgumentException: Illegal bufferSize
> --------------------------------------------------------------------------------
>
> Key: HBASE-28655
> URL: https://issues.apache.org/jira/browse/HBASE-28655
> Project: HBase
> Issue Type: Bug
> Components: HFile, Operability
> Affects Versions: 2.6.0, 3.0.0-beta-1, 2.5.8
> Reporter: Pankaj Kumar
> Assignee: Pankaj Kumar
> Priority: Major
>
> HADOOP-18810 added io.compression.codec.zstd.buffersize in core-default.xml with default value as 0.
> So ZSTD buffer size will be returned as 0 based on core-default.xml,
> {code:java}
> static int getBufferSize(Configuration conf) {
> return conf.getInt(ZSTD_BUFFER_SIZE_KEY,
> conf.getInt(CommonConfigurationKeys.IO_COMPRESSION_CODEC_ZSTD_BUFFER_SIZE_KEY,
> // IO_COMPRESSION_CODEC_ZSTD_BUFFER_SIZE_DEFAULT is 0! We can't allow that.
> ZSTD_BUFFER_SIZE_DEFAULT));
> }
> {code}
> HBASE-26259 added a value check, but got reverted in HBASE-26959.
>
> This issue will also occur during region flush and abort the RegionServer.
>
> TestHFileCompressionZstd and other zstd related test cases are are also failing,
> {code:java}
> java.lang.IllegalArgumentException: Illegal bufferSize
> at org.apache.hadoop.io.compress.CompressorStream.<init>(CompressorStream.java:42)
> at org.apache.hadoop.io.compress.BlockCompressorStream.<init>(BlockCompressorStream.java:56)
> at org.apache.hadoop.hbase.io.compress.aircompressor.ZstdCodec.createOutputStream(ZstdCodec.java:106)
> at org.apache.hadoop.hbase.io.compress.Compression$Algorithm.createPlainCompressionStream(Compression.java:454)
> at org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultEncodingContext.<init>(HFileBlockDefaultEncodingContext.java:99)
> at org.apache.hadoop.hbase.io.hfile.NoOpDataBlockEncoder.newDataBlockEncodingContext(NoOpDataBlockEncoder.java:85)
> at org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.<init>(HFileBlock.java:846)
> at org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.finishInit(HFileWriterImpl.java:304)
> at org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.<init>(HFileWriterImpl.java:185)
> at org.apache.hadoop.hbase.io.hfile.HFile$WriterFactory.create(HFile.java:312)
> at org.apache.hadoop.hbase.io.compress.HFileTestBase.doTest(HFileTestBase.java:73)
> at org.apache.hadoop.hbase.io.compress.aircompressor.TestHFileCompressionZstd.test(TestHFileCompressionZstd.java:54)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)