You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Shankar hiremath <sh...@huawei.com> on 2014/08/01 16:12:35 UTC
Problem reading HFile Trailer by using HFile tool
When I am reading the Hfile using the './hbase org.apache.hadoop.hbase.io.hfile.Hfile' tool (the HFile which is compressed with SNAPPY & encrypted with AES)
I am getting the below error "Problem reading HFile Trailer by using HFile tool"
Is there any problem in the usage of the below command, or is it a bug.
------------------------------------------------------------------------------------------
Version details: Hadoop 2.4.1, HBase 0.98.3
Configuration Details: (enabled Hfile and Wal encryption AES, as below)
hfile.format.version=3
hbase.crypto.keyprovider=org.apache.hadoop.hbase.io.crypto.KeyStoreKeyProvider
hbase.crypto.keyprovider.parameters= jceks:///opt/shankar1/kdc_keytab/hbase.jks?password=shankar@234
We created a table with SNAPPY compression
> Create 't3', {NAME => 'cf1', COMPRESSION => 'SNAPPY'}
> Put 't3','r1','cf1:a','1000'
> Flush 't3'
shankar1@host1:~/DataSight/hbase/bin> ./hbase org.apache.hadoop.hbase.io.hfile.HFile -v -f hdfs://host1:65110/hbase/data/default/t3/337d2996bed579340a702feaa3d3f165/cf1/5817635667d7457989b6d0b0be25dbc4
2014-08-01 19:18:28,368 INFO [main] Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
2014-08-01 19:18:28,504 INFO [main] util.ChecksumType: Checksum using org.apache.hadoop.util.PureJavaCrc32
2014-08-01 19:18:28,506 INFO [main] util.ChecksumType: Checksum can use org.apache.hadoop.util.PureJavaCrc32C
2014-08-01 19:18:28,739 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2014-08-01 19:18:29,082 INFO [main] hdfs.DFSClient: Set dfs.client.block.write.replace-datanode-on-failure.replication to 0
2014-08-01 19:18:29,406 INFO [main] Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS
Scanning -> hdfs://host1:65110/hbase/data/default/t3/337d2996bed579340a702feaa3d3f165/cf1/5817635667d7457989b6d0b0be25dbc4
2014-08-01 19:18:29,409 INFO [main] hdfs.DFSClient: Set dfs.client.block.write.replace-datanode-on-failure.replication to 0
INFO: Watching file:/opt/shankar1/DataSight/hbase/conf/log4j.properties for changes with interval : 60000
2014-08-01 19:18:29,779 ERROR [main] hfile.HFilePrettyPrinter: Error reading hdfs://host1:65110/hbase/data/default/t3/337d2996bed579340a702feaa3d3f165/cf1/5817635667d7457989b6d0b0be25dbc4
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading HFile Trailer from file hdfs://host1:65110/hbase/data/default/t3/337d2996bed579340a702feaa3d3f165/cf1/5817635667d7457989b6d0b0be25dbc4
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:552)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:595)
at org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.processFile(HFilePrettyPrinter.java:217)
at org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.run(HFilePrettyPrinter.java:196)
at org.apache.hadoop.hbase.io.hfile.HFile.main(HFile.java:873)
Caused by: java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native Method)
at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63)
at org.apache.hadoop.io.compress.SnappyCodec.getDecompressorType(SnappyCodec.java:190)
at org.apache.hadoop.io.compress.CodecPool.getDecompressor(CodecPool.java:176)
at org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getDecompressor(Compression.java:336)
at org.apache.hadoop.hbase.io.compress.Compression.decompress(Compression.java:433)
at org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultDecodingContext.prepareDecoding(HFileBlockDefaultDecodingContext.java:91)
at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1522)
at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1314)
at org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlock(HFileBlock.java:1151)
at org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlockWithBlockType(HFileBlock.java:1159)
at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.<init>(HFileReaderV2.java:146)
at org.apache.hadoop.hbase.io.hfile.HFileReaderV3.<init>(HFileReaderV3.java:72)
at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:542)
... 4 more
Scanned kv count -> 0
shankar1@host1:~/DataSight/hbase/bin>
Thanks
-Shankar
[X]
This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!
[X]
Re: Problem reading HFile Trailer by using HFile tool
Posted by Andrew Purtell <ap...@apache.org>.
You have a problem with your environment:
Caused by: java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.
NativeCodeLoader.buildSupportsSnappy()Z
Fix your native Hadoop libraries, or don't use Snappy.
This is not related to encryption.
On Fri, Aug 1, 2014 at 7:12 AM, Shankar hiremath <
shankar.hiremath@huawei.com> wrote:
> When I am reading the Hfile using the './hbase
> org.apache.hadoop.hbase.io.hfile.Hfile' tool (the HFile which is compressed
> with SNAPPY & encrypted with AES)
> I am getting the below error "Problem reading HFile Trailer by using HFile
> tool"
>
> Is there any problem in the usage of the below command, or is it a bug.
>
>
> ------------------------------------------------------------------------------------------
>
> Version details: Hadoop 2.4.1, HBase 0.98.3
>
> Configuration Details: (enabled Hfile and Wal encryption AES, as below)
> hfile.format.version=3
>
> hbase.crypto.keyprovider=org.apache.hadoop.hbase.io.crypto.KeyStoreKeyProvider
> hbase.crypto.keyprovider.parameters=
> jceks:///opt/shankar1/kdc_keytab/hbase.jks?password=shankar@234
>
> We created a table with SNAPPY compression
> > Create 't3', {NAME => 'cf1', COMPRESSION => 'SNAPPY'}
> > Put 't3','r1','cf1:a','1000'
> > Flush 't3'
>
> shankar1@host1:~/DataSight/hbase/bin> ./hbase
> org.apache.hadoop.hbase.io.hfile.HFile -v -f
> hdfs://host1:65110/hbase/data/default/t3/337d2996bed579340a702feaa3d3f165/cf1/5817635667d7457989b6d0b0be25dbc4
> 2014-08-01 19:18:28,368 INFO [main] Configuration.deprecation:
> hadoop.native.lib is deprecated. Instead, use io.native.lib.available
> 2014-08-01 19:18:28,504 INFO [main] util.ChecksumType: Checksum using
> org.apache.hadoop.util.PureJavaCrc32
> 2014-08-01 19:18:28,506 INFO [main] util.ChecksumType: Checksum can use
> org.apache.hadoop.util.PureJavaCrc32C
> 2014-08-01 19:18:28,739 WARN [main] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> 2014-08-01 19:18:29,082 INFO [main] hdfs.DFSClient: Set
> dfs.client.block.write.replace-datanode-on-failure.replication to 0
> 2014-08-01 19:18:29,406 INFO [main] Configuration.deprecation:
> fs.default.name is deprecated. Instead, use fs.defaultFS
> Scanning ->
> hdfs://host1:65110/hbase/data/default/t3/337d2996bed579340a702feaa3d3f165/cf1/5817635667d7457989b6d0b0be25dbc4
> 2014-08-01 19:18:29,409 INFO [main] hdfs.DFSClient: Set
> dfs.client.block.write.replace-datanode-on-failure.replication to 0
> INFO: Watching file:/opt/shankar1/DataSight/hbase/conf/log4j.properties
> for changes with interval : 60000
> 2014-08-01 19:18:29,779 ERROR [main] hfile.HFilePrettyPrinter: Error
> reading
> hdfs://host1:65110/hbase/data/default/t3/337d2996bed579340a702feaa3d3f165/cf1/5817635667d7457989b6d0b0be25dbc4
> org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading
> HFile Trailer from file
> hdfs://host1:65110/hbase/data/default/t3/337d2996bed579340a702feaa3d3f165/cf1/5817635667d7457989b6d0b0be25dbc4
> at
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:552)
> at
> org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:595)
> at
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.processFile(HFilePrettyPrinter.java:217)
> at
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.run(HFilePrettyPrinter.java:196)
> at org.apache.hadoop.hbase.io.hfile.HFile.main(HFile.java:873)
> Caused by: java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
> at
> org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native Method)
> at
> org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63)
> at
> org.apache.hadoop.io.compress.SnappyCodec.getDecompressorType(SnappyCodec.java:190)
> at
> org.apache.hadoop.io.compress.CodecPool.getDecompressor(CodecPool.java:176)
> at
> org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getDecompressor(Compression.java:336)
> at
> org.apache.hadoop.hbase.io.compress.Compression.decompress(Compression.java:433)
> at
> org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultDecodingContext.prepareDecoding(HFileBlockDefaultDecodingContext.java:91)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1522)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1314)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlock(HFileBlock.java:1151)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader$1.nextBlockWithBlockType(HFileBlock.java:1159)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.<init>(HFileReaderV2.java:146)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV3.<init>(HFileReaderV3.java:72)
> at
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:542)
> ... 4 more
> Scanned kv count -> 0
> shankar1@host1:~/DataSight/hbase/bin>
>
> Thanks
> -Shankar
>
> [X]
> This e-mail and its attachments contain confidential information from
> HUAWEI, which is intended only for the person or entity whose address is
> listed above. Any use of the information contained herein in any way
> (including, but not limited to, total or partial disclosure, reproduction,
> or dissemination) by persons other than the intended recipient(s) is
> prohibited. If you receive this e-mail in error, please notify the sender
> by phone or email immediately and delete it!
> [X]
>
>
>
>
>
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)