[jira] [Created] (HBASE-27706) Possible Zstd incompatibility

["Frens Jan Rumph (Jira)" <ji...@apache.org>]

Frens Jan Rumph created HBASE-27706:
---------------------------------------

             Summary: Possible Zstd incompatibility
                 Key: HBASE-27706
                 URL: https://issues.apache.org/jira/browse/HBASE-27706
             Project: HBase
          Issue Type: Bug
          Components: compatibility
    Affects Versions: 2.5.3
            Reporter: Frens Jan Rumph


 

We're in the process of upgrading a HBase installation from 2.2.4 to 2.5.3. We're currently using Zstd compression from our Hadoop installation. Due to some other class path issues (Netty issues in relation to the async WAL provider), we would like to remove Hadoop from the class path.

However, using the Zstd compression from HBase (which uses [https://github.com/luben/zstd-jni]) we seem to hit some incompatibility. When restarting a node to use this implementation we had errors like the following:
{code:java}
2023-03-10 16:33:01,925 WARN  [RS_OPEN_REGION-regionserver/n2:16020-0] handler.AssignRegionHandler: Failed to open region NAMESPACE:TABLE,,1673888962751.cdb726dad4eaabf765969f195e91c737., will report to master
java.io.IOException: java.io.IOException: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading data index and meta index from file hdfs://CLUSTER/hbase/data/NAMESPACE/TABLE/cdb726dad4eaabf765969f195e91c737/e/aea6eddaa8ee476197d064a4b4c345b9
        at org.apache.hadoop.hbase.regionserver.HRegion.initializeStores(HRegion.java:1148)
        at org.apache.hadoop.hbase.regionserver.HRegion.initializeStores(HRegion.java:1091)
        at org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:994)
        at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:941)
        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7228)
        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegionFromTableDir(HRegion.java:7183)
        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7159)
        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7118)
        at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7074)
        at org.apache.hadoop.hbase.regionserver.handler.AssignRegionHandler.process(AssignRegionHandler.java:147)
        at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:100)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.IOException: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading data index and meta index from file hdfs://CLUSTER/hbase/data/NAMESPACE/TABLE/cdb726dad4eaabf765969f195e91c737/e/aea6eddaa8ee476197d064a4b4c345b9
        at org.apache.hadoop.hbase.regionserver.StoreEngine.openStoreFiles(StoreEngine.java:288)
        at org.apache.hadoop.hbase.regionserver.StoreEngine.initialize(StoreEngine.java:338)
        at org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:297)
        at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:6359)
        at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1114)
        at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1111)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        ... 3 more
Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading data index and meta index from file hdfs://CLUSTER/hbase/data/NAMESPACE/TABLE/cdb726dad4eaabf765969f195e91c737/e/aea6eddaa8ee476197d064a4b4c345b9
        at org.apache.hadoop.hbase.io.hfile.HFileInfo.initMetaAndIndex(HFileInfo.java:392)
        at org.apache.hadoop.hbase.regionserver.HStoreFile.open(HStoreFile.java:394)
        at org.apache.hadoop.hbase.regionserver.HStoreFile.initReader(HStoreFile.java:518)
        at org.apache.hadoop.hbase.regionserver.StoreEngine.createStoreFileAndReader(StoreEngine.java:225)
        at org.apache.hadoop.hbase.regionserver.StoreEngine.lambda$openStoreFiles$0(StoreEngine.java:266)
        ... 6 more
Caused by: java.io.IOException: Premature EOF from inputStream, but still need 2883 bytes
        at org.apache.hadoop.hbase.io.util.BlockIOUtils.readFullyWithHeapBuffer(BlockIOUtils.java:153)
        at org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultDecodingContext.prepareDecoding(HFileBlockDefaultDecodingContext.java:104)
        at org.apache.hadoop.hbase.io.hfile.HFileBlock.unpack(HFileBlock.java:644)
        at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl$1.nextBlock(HFileBlock.java:1397)
        at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl$1.nextBlockWithBlockType(HFileBlock.java:1407)
        at org.apache.hadoop.hbase.io.hfile.HFileInfo.initMetaAndIndex(HFileInfo.java:365)
        ... 10 more {code}
I've been able to reproduce the issue with something like:
{code:java}
Configuration conf = HBaseConfiguration.create();
conf.set("hbase.io.compress.zstd.codec", "org.apache.hadoop.hbase.io.compress.zstd.ZstdCodec");
HFileSystem fs = (HFileSystem) HFileSystem.get(conf);
HFile.createReader(fs, new Path(...), conf); {code}
with a file from HDFS that was created with the native compressor from Hadoop.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)