You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Pankaj Kumar (Jira)" <ji...@apache.org> on 2024/07/23 14:23:00 UTC

[jira] [Updated] (HBASE-28655) TestHFileCompressionZstd fails with IllegalArgumentException: Illegal bufferSize

     [ https://issues.apache.org/jira/browse/HBASE-28655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Pankaj Kumar updated HBASE-28655:
---------------------------------
    Description: 
HADOOP-18810 added io.compression.codec.zstd.buffersize in core-default.xml with default value as 0.

So ZSTD buffer size will be returned as 0 based on core-default.xml,
{code:java}
  static int getBufferSize(Configuration conf) {
    return conf.getInt(ZSTD_BUFFER_SIZE_KEY,
      conf.getInt(CommonConfigurationKeys.IO_COMPRESSION_CODEC_ZSTD_BUFFER_SIZE_KEY,
        // IO_COMPRESSION_CODEC_ZSTD_BUFFER_SIZE_DEFAULT is 0! We can't allow that.
        ZSTD_BUFFER_SIZE_DEFAULT));
  }
{code}
HBASE-26259 added a value check, but got reverted in HBASE-26959.

 

This issue will also occur during region flush and abort the RegionServer.

 

TestHFileCompressionZstd and other zstd related test cases are are also failing,
{code:java}
java.lang.IllegalArgumentException: Illegal bufferSize
	at org.apache.hadoop.io.compress.CompressorStream.<init>(CompressorStream.java:42)
	at org.apache.hadoop.io.compress.BlockCompressorStream.<init>(BlockCompressorStream.java:56)
	at org.apache.hadoop.hbase.io.compress.aircompressor.ZstdCodec.createOutputStream(ZstdCodec.java:106)
	at org.apache.hadoop.hbase.io.compress.Compression$Algorithm.createPlainCompressionStream(Compression.java:454)
	at org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultEncodingContext.<init>(HFileBlockDefaultEncodingContext.java:99)
	at org.apache.hadoop.hbase.io.hfile.NoOpDataBlockEncoder.newDataBlockEncodingContext(NoOpDataBlockEncoder.java:85)
	at org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.<init>(HFileBlock.java:846)
	at org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.finishInit(HFileWriterImpl.java:304)
	at org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.<init>(HFileWriterImpl.java:185)
	at org.apache.hadoop.hbase.io.hfile.HFile$WriterFactory.create(HFile.java:312)
	at org.apache.hadoop.hbase.io.compress.HFileTestBase.doTest(HFileTestBase.java:73)
	at org.apache.hadoop.hbase.io.compress.aircompressor.TestHFileCompressionZstd.test(TestHFileCompressionZstd.java:54)
{code}

  was:
HADOOP-18810 added io.compression.codec.zstd.buffersize in core-default.xml with default value as 0. 

So ZSTD buffer size will be returned as 0 based on core-default.xml,
{code}
  static int getBufferSize(Configuration conf) {
    return conf.getInt(ZSTD_BUFFER_SIZE_KEY,
      conf.getInt(CommonConfigurationKeys.IO_COMPRESSION_CODEC_ZSTD_BUFFER_SIZE_KEY,
        // IO_COMPRESSION_CODEC_ZSTD_BUFFER_SIZE_DEFAULT is 0! We can't allow that.
        ZSTD_BUFFER_SIZE_DEFAULT));
  }
{code}

HBASE-26259 added a value check, but got reverted in HBASE-26959.

TestHFileCompressionZstd and other zstd related test cases are failing,
{code}
java.lang.IllegalArgumentException: Illegal bufferSize
	at org.apache.hadoop.io.compress.CompressorStream.<init>(CompressorStream.java:42)
	at org.apache.hadoop.io.compress.BlockCompressorStream.<init>(BlockCompressorStream.java:56)
	at org.apache.hadoop.hbase.io.compress.aircompressor.ZstdCodec.createOutputStream(ZstdCodec.java:106)
	at org.apache.hadoop.hbase.io.compress.Compression$Algorithm.createPlainCompressionStream(Compression.java:454)
	at org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultEncodingContext.<init>(HFileBlockDefaultEncodingContext.java:99)
	at org.apache.hadoop.hbase.io.hfile.NoOpDataBlockEncoder.newDataBlockEncodingContext(NoOpDataBlockEncoder.java:85)
	at org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.<init>(HFileBlock.java:846)
	at org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.finishInit(HFileWriterImpl.java:304)
	at org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.<init>(HFileWriterImpl.java:185)
	at org.apache.hadoop.hbase.io.hfile.HFile$WriterFactory.create(HFile.java:312)
	at org.apache.hadoop.hbase.io.compress.HFileTestBase.doTest(HFileTestBase.java:73)
	at org.apache.hadoop.hbase.io.compress.aircompressor.TestHFileCompressionZstd.test(TestHFileCompressionZstd.java:54)
{code}


> TestHFileCompressionZstd fails with IllegalArgumentException: Illegal bufferSize
> --------------------------------------------------------------------------------
>
>                 Key: HBASE-28655
>                 URL: https://issues.apache.org/jira/browse/HBASE-28655
>             Project: HBase
>          Issue Type: Bug
>          Components: HFile, Operability
>    Affects Versions: 2.6.0, 3.0.0-beta-1, 2.5.8
>            Reporter: Pankaj Kumar
>            Assignee: Pankaj Kumar
>            Priority: Major
>
> HADOOP-18810 added io.compression.codec.zstd.buffersize in core-default.xml with default value as 0.
> So ZSTD buffer size will be returned as 0 based on core-default.xml,
> {code:java}
>   static int getBufferSize(Configuration conf) {
>     return conf.getInt(ZSTD_BUFFER_SIZE_KEY,
>       conf.getInt(CommonConfigurationKeys.IO_COMPRESSION_CODEC_ZSTD_BUFFER_SIZE_KEY,
>         // IO_COMPRESSION_CODEC_ZSTD_BUFFER_SIZE_DEFAULT is 0! We can't allow that.
>         ZSTD_BUFFER_SIZE_DEFAULT));
>   }
> {code}
> HBASE-26259 added a value check, but got reverted in HBASE-26959.
>  
> This issue will also occur during region flush and abort the RegionServer.
>  
> TestHFileCompressionZstd and other zstd related test cases are are also failing,
> {code:java}
> java.lang.IllegalArgumentException: Illegal bufferSize
> 	at org.apache.hadoop.io.compress.CompressorStream.<init>(CompressorStream.java:42)
> 	at org.apache.hadoop.io.compress.BlockCompressorStream.<init>(BlockCompressorStream.java:56)
> 	at org.apache.hadoop.hbase.io.compress.aircompressor.ZstdCodec.createOutputStream(ZstdCodec.java:106)
> 	at org.apache.hadoop.hbase.io.compress.Compression$Algorithm.createPlainCompressionStream(Compression.java:454)
> 	at org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultEncodingContext.<init>(HFileBlockDefaultEncodingContext.java:99)
> 	at org.apache.hadoop.hbase.io.hfile.NoOpDataBlockEncoder.newDataBlockEncodingContext(NoOpDataBlockEncoder.java:85)
> 	at org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.<init>(HFileBlock.java:846)
> 	at org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.finishInit(HFileWriterImpl.java:304)
> 	at org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.<init>(HFileWriterImpl.java:185)
> 	at org.apache.hadoop.hbase.io.hfile.HFile$WriterFactory.create(HFile.java:312)
> 	at org.apache.hadoop.hbase.io.compress.HFileTestBase.doTest(HFileTestBase.java:73)
> 	at org.apache.hadoop.hbase.io.compress.aircompressor.TestHFileCompressionZstd.test(TestHFileCompressionZstd.java:54)
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)