You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Hanish Bansal <ha...@gmail.com> on 2014/07/11 12:21:30 UTC
Snappy compression not working with HBase 0.98.3
Hi All,
Recently i have upgraded HBase environment from 0.94 to 0.98.3. Now trying
to use snappy compression with it.
I have installed snappy library as per guide mentioned in
https://hbase.apache.org/book/snappy.compression.html
When i am creating a table with snappy compression enabled, i am getting
below error:
hbase(main):001:0> create 'test', {NAME=>'cf1', COMPRESSION=>'SNAPPY'}
2014-07-08 20:06:33,265 WARN [main] util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
ERROR: java.io.IOException: Compression algorithm 'snappy' previously
failed test.
at
org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:85)
at
org.apache.hadoop.hbase.master.HMaster.checkCompression(HMaster.java:1774)
at
org.apache.hadoop.hbase.master.HMaster.checkCompression(HMaster.java:1767)
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1749)
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1784)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:40470)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2012)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
at
org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:73)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Please let me know if anyone aware of this issue.
--
*Thanks & Regards*
*Hanish Bansal*
RE: Snappy compression not working with HBase 0.98.3
Posted by Kashif Jawed Siddiqui <ka...@huawei.com>.
Add hadoop\lib\native to the HBASE CLASSPATH
The $HADOOP_HOME\lib\native contains the snappy libs
Thumbs Up !
KASHIF
-----Original Message-----
From: Hanish Bansal [mailto:hanish.bansal.agarwal@gmail.com]
Sent: 11 July 2014 16:08
To: user@hbase.apache.org
Subject: Re: Snappy compression not working with HBase 0.98.3
We are using hbase 0.98.3 with hadoop 2.4.0.
Run a compression test using tool, getting below error:
[root@IMPETUS-I0141 hbase-0.98.3-hadoop2]# bin/hbase org.apache.hadoop.hbase.util.CompressionTest file:///tmp/test.txt snappy
2014-07-11 16:05:10,572 INFO [main] Configuration.deprecation:
hadoop.native.lib is deprecated. Instead, use io.native.lib.available
2014-07-11 16:05:11,006 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2014-07-11 16:05:11,241 INFO [main] util.ChecksumType: Checksum using
org.apache.hadoop.util.PureJavaCrc32
2014-07-11 16:05:11,242 INFO [main] util.ChecksumType: Checksum can use org.apache.hadoop.util.PureJavaCrc32C
Exception in thread "main" java.lang.UnsatisfiedLinkError:
org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native
Method)
at
org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:62)
at
org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:131)
at
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:147)
at
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:162)
at
org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:310)
at
org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultEncodingContext.<init>(HFileBlockDefaultEncodingContext.java:92)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.<init>(HFileBlock.java:690)
at
org.apache.hadoop.hbase.io.hfile.HFileWriterV2.finishInit(HFileWriterV2.java:117)
at
org.apache.hadoop.hbase.io.hfile.HFileWriterV2.<init>(HFileWriterV2.java:109)
at
org.apache.hadoop.hbase.io.hfile.HFileWriterV2$WriterFactoryV2.createWriter(HFileWriterV2.java:97)
at
org.apache.hadoop.hbase.io.hfile.HFile$WriterFactory.create(HFile.java:393)
at
org.apache.hadoop.hbase.util.CompressionTest.doSmokeTest(CompressionTest.java:118)
at
org.apache.hadoop.hbase.util.CompressionTest.main(CompressionTest.java:148)
On Fri, Jul 11, 2014 at 10:21 AM, Hanish Bansal < hanish.bansal.agarwal@gmail.com> wrote:
> Hi All,
>
> Recently i have upgraded HBase environment from 0.94 to 0.98.3. Now
> trying to use snappy compression with it.
>
> I have installed snappy library as per guide mentioned in
> https://hbase.apache.org/book/snappy.compression.html
>
> When i am creating a table with snappy compression enabled, i am
> getting below error:
>
>
> hbase(main):001:0> create 'test', {NAME=>'cf1', COMPRESSION=>'SNAPPY'}
> 2014-07-08 20:06:33,265 WARN [main] util.NativeCodeLoader: Unable to
> load native-hadoop library for your platform... using builtin-java
> classes where applicable
>
> ERROR: java.io.IOException: Compression algorithm 'snappy' previously
> failed test.
> at
> org.apache.hadoop.hbase.util.CompressionTest.testCompression(Compressi
> onTest.java:85)
> at
> org.apache.hadoop.hbase.master.HMaster.checkCompression(HMaster.java:1
> 774)
> at
> org.apache.hadoop.hbase.master.HMaster.checkCompression(HMaster.java:1
> 767) at
> org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1749)
> at
> org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1784)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$
> 2.callBlockingMethod(MasterProtos.java:40470)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2012)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
> at
> org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.ja
> va:73) at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471
> ) at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j
> ava:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.
> java:615) at java.lang.Thread.run(Thread.java:722)
>
>
> Please let me know if anyone aware of this issue.
>
> --
> *Thanks & Regards*
> *Hanish Bansal*
>
--
*Thanks & Regards*
*Hanish Bansal*
Re: Snappy compression not working with HBase 0.98.3
Posted by Hanish Bansal <ha...@gmail.com>.
Hi all,
Thanks for reply.
Finally it worked by following post
http://www.spaggiari.org/index.php/hbase/how-to-install-snappy-with-1#.U8Pui-9ZuZY
.
Issue was:
By default hadoop-2 comes with 32 bit native library and without native
support so i need to compile source code of hadoop for 64 bit architecture
with native support. After that it worked fine.
On Mon, Jul 14, 2014 at 3:09 PM, Stack <st...@duboce.net> wrote:
> On Mon, Jul 14, 2014 at 7:49 AM, Hanish Bansal <
> hanish.bansal.agarwal@gmail.com> wrote:
>
> > Hi All,
> >
> > We have tried below things:
> >
> > 1. Pointed HBase to hadoop and snappy libraries which hadoop holds :
> >
> > export HBASE_LIBRARY_PATH=/pathtoyourhadoop/lib/native/Linux-amd64-64
> >
> > As hadoop holds hadoop and snappy library, it should work. But it didn't.
> >
> > 2. Copied libhadoop.so and libsnappy.so to hbase native library folder
> > at $HBASE_HOME/lib/native/Linux-amd64-64/.
> >
> > It also didn't work.
> >
> > *Run a compression test using tool, getting below error:*
> >
> > [root@IMPETUS-I0141 hbase-0.98.3-hadoop2]# bin/hbase
> > org.apache.hadoop.hbase.util.
> >
> > CompressionTest file:///tmp/test.txt snappy
> > 2014-07-11 16:05:10,572 INFO [main] Configuration.deprecation:
> > hadoop.native.lib is deprecated. Instead, use io.native.lib.available
> > 2014-07-11 16:05:11,006 WARN [main] util.NativeCodeLoader: Unable to
> load
> > native-hadoop library for your platform... using builtin-java classes
> where
> > applicable
> > 2014-07-11 16:05:11,241 INFO [main] util.ChecksumType: Checksum using
> > org.apache.hadoop.util.PureJavaCrc32
> > 2014-07-11 16:05:11,242 INFO [main] util.ChecksumType: Checksum can use
> > org.apache.hadoop.util.PureJavaCrc32C
> > Exception in thread "main" java.lang.UnsatisfiedLinkError:
> > org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
> > at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native
> > Method)
> > at
> >
> >
> org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:62)
> > at
> >
> >
> org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:131)
> > at
> > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:147)
> > at
> > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:162)
> > at
> >
> >
> org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:310)
> > at
> >
> >
> org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultEncodingContext.<init>(HFileBlockDefaultEncodingContext.java:92)
> > at
> >
> >
> org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.<init>(HFileBlock.java:690)
> > at
> >
> >
> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.finishInit(HFileWriterV2.java:117)
> > at
> >
> >
> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.<init>(HFileWriterV2.java:109)
> > at
> >
> >
> org.apache.hadoop.hbase.io.hfile.HFileWriterV2$WriterFactoryV2.createWriter(HFileWriterV2.java:97)
> > at
> >
> org.apache.hadoop.hbase.io.hfile.HFile$WriterFactory.create(HFile.java:393)
> > at
> >
> >
> org.apache.hadoop.hbase.util.CompressionTest.doSmokeTest(CompressionTest.java:118)
> > at
> >
> org.apache.hadoop.hbase.util.CompressionTest.main(CompressionTest.java:148)
> >
> > Everything was working fine with hbase-0.94.5 as well as hbase-0.96.1.
> >
> >
> >
> It is not finding your native lib. Did you do this on the command line or
> in the hbase-env.sh script?
>
> export HBASE_LIBRARY_PATH=/pathtoyourhadoop/lib/native/Linux-amd64-64
>
> Did you change 'pathtoyourhadoop' to point at actual library? Is the lib
> there? Is your architecture Linux-amd64-64 for sure? (See the text I
> posted last night for how to figure what your system is)
>
> St.Ack
>
--
*Thanks & Regards*
*Hanish Bansal*
Re: Snappy compression not working with HBase 0.98.3
Posted by Stack <st...@duboce.net>.
On Mon, Jul 14, 2014 at 7:49 AM, Hanish Bansal <
hanish.bansal.agarwal@gmail.com> wrote:
> Hi All,
>
> We have tried below things:
>
> 1. Pointed HBase to hadoop and snappy libraries which hadoop holds :
>
> export HBASE_LIBRARY_PATH=/pathtoyourhadoop/lib/native/Linux-amd64-64
>
> As hadoop holds hadoop and snappy library, it should work. But it didn't.
>
> 2. Copied libhadoop.so and libsnappy.so to hbase native library folder
> at $HBASE_HOME/lib/native/Linux-amd64-64/.
>
> It also didn't work.
>
> *Run a compression test using tool, getting below error:*
>
> [root@IMPETUS-I0141 hbase-0.98.3-hadoop2]# bin/hbase
> org.apache.hadoop.hbase.util.
>
> CompressionTest file:///tmp/test.txt snappy
> 2014-07-11 16:05:10,572 INFO [main] Configuration.deprecation:
> hadoop.native.lib is deprecated. Instead, use io.native.lib.available
> 2014-07-11 16:05:11,006 WARN [main] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> 2014-07-11 16:05:11,241 INFO [main] util.ChecksumType: Checksum using
> org.apache.hadoop.util.PureJavaCrc32
> 2014-07-11 16:05:11,242 INFO [main] util.ChecksumType: Checksum can use
> org.apache.hadoop.util.PureJavaCrc32C
> Exception in thread "main" java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
> at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native
> Method)
> at
>
> org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:62)
> at
>
> org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:131)
> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:147)
> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:162)
> at
>
> org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:310)
> at
>
> org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultEncodingContext.<init>(HFileBlockDefaultEncodingContext.java:92)
> at
>
> org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.<init>(HFileBlock.java:690)
> at
>
> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.finishInit(HFileWriterV2.java:117)
> at
>
> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.<init>(HFileWriterV2.java:109)
> at
>
> org.apache.hadoop.hbase.io.hfile.HFileWriterV2$WriterFactoryV2.createWriter(HFileWriterV2.java:97)
> at
> org.apache.hadoop.hbase.io.hfile.HFile$WriterFactory.create(HFile.java:393)
> at
>
> org.apache.hadoop.hbase.util.CompressionTest.doSmokeTest(CompressionTest.java:118)
> at
> org.apache.hadoop.hbase.util.CompressionTest.main(CompressionTest.java:148)
>
> Everything was working fine with hbase-0.94.5 as well as hbase-0.96.1.
>
>
>
It is not finding your native lib. Did you do this on the command line or
in the hbase-env.sh script?
export HBASE_LIBRARY_PATH=/pathtoyourhadoop/lib/native/Linux-amd64-64
Did you change 'pathtoyourhadoop' to point at actual library? Is the lib
there? Is your architecture Linux-amd64-64 for sure? (See the text I
posted last night for how to figure what your system is)
St.Ack
Re: Snappy compression not working with HBase 0.98.3
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Hi Hanish,
I don't know if that will help but I wrote that some time ago:
http://www.spaggiari.org/index.php/hbase/how-to-install-snappy-with-1#.U8Pui-9ZuZY
It's most probably the same steps with 0.98. When I need to test I simply
copy the lib/native folder tha tI built for 0.96 to the 0.98 folder...
JM
2014-07-14 10:49 GMT-04:00 Hanish Bansal <ha...@gmail.com>:
> Hi All,
>
> We have tried below things:
>
> 1. Pointed HBase to hadoop and snappy libraries which hadoop holds :
>
> export HBASE_LIBRARY_PATH=/pathtoyourhadoop/lib/native/Linux-amd64-64
>
> As hadoop holds hadoop and snappy library, it should work. But it didn't.
>
> 2. Copied libhadoop.so and libsnappy.so to hbase native library folder
> at $HBASE_HOME/lib/native/Linux-amd64-64/.
>
> It also didn't work.
>
> *Run a compression test using tool, getting below error:*
>
> [root@IMPETUS-I0141 hbase-0.98.3-hadoop2]# bin/hbase
> org.apache.hadoop.hbase.util.
>
> CompressionTest file:///tmp/test.txt snappy
> 2014-07-11 16:05:10,572 INFO [main] Configuration.deprecation:
> hadoop.native.lib is deprecated. Instead, use io.native.lib.available
> 2014-07-11 16:05:11,006 WARN [main] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> 2014-07-11 16:05:11,241 INFO [main] util.ChecksumType: Checksum using
> org.apache.hadoop.util.PureJavaCrc32
> 2014-07-11 16:05:11,242 INFO [main] util.ChecksumType: Checksum can use
> org.apache.hadoop.util.PureJavaCrc32C
> Exception in thread "main" java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
> at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native
> Method)
> at
>
> org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:62)
> at
>
> org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:131)
> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:147)
> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:162)
> at
>
> org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:310)
> at
>
> org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultEncodingContext.<init>(HFileBlockDefaultEncodingContext.java:92)
> at
>
> org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.<init>(HFileBlock.java:690)
> at
>
> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.finishInit(HFileWriterV2.java:117)
> at
>
> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.<init>(HFileWriterV2.java:109)
> at
>
> org.apache.hadoop.hbase.io.hfile.HFileWriterV2$WriterFactoryV2.createWriter(HFileWriterV2.java:97)
> at
> org.apache.hadoop.hbase.io.hfile.HFile$WriterFactory.create(HFile.java:393)
> at
>
> org.apache.hadoop.hbase.util.CompressionTest.doSmokeTest(CompressionTest.java:118)
> at
> org.apache.hadoop.hbase.util.CompressionTest.main(CompressionTest.java:148)
>
> Everything was working fine with hbase-0.94.5 as well as hbase-0.96.1.
>
>
>
>
>
> On Mon, Jul 14, 2014 at 5:40 AM, Stack <st...@duboce.net> wrote:
>
> > On Sun, Jul 13, 2014 at 10:28 PM, Esteban Gutierrez <
> esteban@cloudera.com>
> > wrote:
> >
> > > Hello Ankit,
> > >
> > > The only reason the test can fail in the master is that the snappy
> > natives
> > > libraries are not installed correctly . Have you tried to run the
> > > compression test (hbase org.apache.hadoop.hbase.util.CompressionTest
> > > file:///tmp snappy) in the master? does it works? If it works correctly
> > > then you only need to restart the HBase masters in order to get it
> > working.
> > >
> > > cheers,
> > > esteban.
> > >
> > >
> > >
> > >
> > What Esteban says.
> >
> > We added a little bit more on how hbase finds native libs to the doc but
> > have not pushed it out to the website. Perhaps it will help in this case
> > (pardon the formatting):
> >
> > <note xml:id="hbase.native.platform"><title>On the location of
> native
> > libraries</title>
> >
> > <para>Hadoop looks in <filename>lib/native</filename> for .so
> > files. HBase looks in
> > <filename>lib/native/PLATFORM</filename>. See the
> > <command>bin/hbase</command>.
> > View the file and look for <varname>native</varname>. See
> how
> > we
> > do the work to find out what platform we are running on
> > running a little java program
> > <classname>org.apache.hadoop.util.PlatformName</classname>
> to
> > figure it out.
> > We'll then add <filename>./lib/native/PLATFORM</filename> to
> > the
> > <varname>LD_LIBRARY_PATH</varname> environment for when the
> > JVM starts.
> > The JVM will look in here (as well as in any other dirs
> > specified on LD_LIBRARY_PATH)
> > for codec native libs. If you are unable to figure your
> > 'platform', do:
> > <programlisting>$ ./bin/hbase
> > org.apache.hadoop.util.PlatformName</programlisting>.
> > An example platform would be
> > <varname>Linux-amd64-64</varname>.
> > </para>
> >
> > </note>
> >
> > St.Ack
> >
>
>
>
> --
> *Thanks & Regards*
> *Hanish Bansal*
>
Re: Snappy compression not working with HBase 0.98.3
Posted by Hanish Bansal <ha...@gmail.com>.
Hi All,
We have tried below things:
1. Pointed HBase to hadoop and snappy libraries which hadoop holds :
export HBASE_LIBRARY_PATH=/pathtoyourhadoop/lib/native/Linux-amd64-64
As hadoop holds hadoop and snappy library, it should work. But it didn't.
2. Copied libhadoop.so and libsnappy.so to hbase native library folder
at $HBASE_HOME/lib/native/Linux-amd64-64/.
It also didn't work.
*Run a compression test using tool, getting below error:*
[root@IMPETUS-I0141 hbase-0.98.3-hadoop2]# bin/hbase
org.apache.hadoop.hbase.util.
CompressionTest file:///tmp/test.txt snappy
2014-07-11 16:05:10,572 INFO [main] Configuration.deprecation:
hadoop.native.lib is deprecated. Instead, use io.native.lib.available
2014-07-11 16:05:11,006 WARN [main] util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
2014-07-11 16:05:11,241 INFO [main] util.ChecksumType: Checksum using
org.apache.hadoop.util.PureJavaCrc32
2014-07-11 16:05:11,242 INFO [main] util.ChecksumType: Checksum can use
org.apache.hadoop.util.PureJavaCrc32C
Exception in thread "main" java.lang.UnsatisfiedLinkError:
org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native
Method)
at
org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:62)
at
org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:131)
at
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:147)
at
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:162)
at
org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:310)
at
org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultEncodingContext.<init>(HFileBlockDefaultEncodingContext.java:92)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.<init>(HFileBlock.java:690)
at
org.apache.hadoop.hbase.io.hfile.HFileWriterV2.finishInit(HFileWriterV2.java:117)
at
org.apache.hadoop.hbase.io.hfile.HFileWriterV2.<init>(HFileWriterV2.java:109)
at
org.apache.hadoop.hbase.io.hfile.HFileWriterV2$WriterFactoryV2.createWriter(HFileWriterV2.java:97)
at
org.apache.hadoop.hbase.io.hfile.HFile$WriterFactory.create(HFile.java:393)
at
org.apache.hadoop.hbase.util.CompressionTest.doSmokeTest(CompressionTest.java:118)
at
org.apache.hadoop.hbase.util.CompressionTest.main(CompressionTest.java:148)
Everything was working fine with hbase-0.94.5 as well as hbase-0.96.1.
On Mon, Jul 14, 2014 at 5:40 AM, Stack <st...@duboce.net> wrote:
> On Sun, Jul 13, 2014 at 10:28 PM, Esteban Gutierrez <es...@cloudera.com>
> wrote:
>
> > Hello Ankit,
> >
> > The only reason the test can fail in the master is that the snappy
> natives
> > libraries are not installed correctly . Have you tried to run the
> > compression test (hbase org.apache.hadoop.hbase.util.CompressionTest
> > file:///tmp snappy) in the master? does it works? If it works correctly
> > then you only need to restart the HBase masters in order to get it
> working.
> >
> > cheers,
> > esteban.
> >
> >
> >
> >
> What Esteban says.
>
> We added a little bit more on how hbase finds native libs to the doc but
> have not pushed it out to the website. Perhaps it will help in this case
> (pardon the formatting):
>
> <note xml:id="hbase.native.platform"><title>On the location of native
> libraries</title>
>
> <para>Hadoop looks in <filename>lib/native</filename> for .so
> files. HBase looks in
> <filename>lib/native/PLATFORM</filename>. See the
> <command>bin/hbase</command>.
> View the file and look for <varname>native</varname>. See how
> we
> do the work to find out what platform we are running on
> running a little java program
> <classname>org.apache.hadoop.util.PlatformName</classname> to
> figure it out.
> We'll then add <filename>./lib/native/PLATFORM</filename> to
> the
> <varname>LD_LIBRARY_PATH</varname> environment for when the
> JVM starts.
> The JVM will look in here (as well as in any other dirs
> specified on LD_LIBRARY_PATH)
> for codec native libs. If you are unable to figure your
> 'platform', do:
> <programlisting>$ ./bin/hbase
> org.apache.hadoop.util.PlatformName</programlisting>.
> An example platform would be
> <varname>Linux-amd64-64</varname>.
> </para>
>
> </note>
>
> St.Ack
>
--
*Thanks & Regards*
*Hanish Bansal*
Re: Snappy compression not working with HBase 0.98.3
Posted by Stack <st...@duboce.net>.
On Sun, Jul 13, 2014 at 10:28 PM, Esteban Gutierrez <es...@cloudera.com>
wrote:
> Hello Ankit,
>
> The only reason the test can fail in the master is that the snappy natives
> libraries are not installed correctly . Have you tried to run the
> compression test (hbase org.apache.hadoop.hbase.util.CompressionTest
> file:///tmp snappy) in the master? does it works? If it works correctly
> then you only need to restart the HBase masters in order to get it working.
>
> cheers,
> esteban.
>
>
>
>
What Esteban says.
We added a little bit more on how hbase finds native libs to the doc but
have not pushed it out to the website. Perhaps it will help in this case
(pardon the formatting):
<note xml:id="hbase.native.platform"><title>On the location of native
libraries</title>
<para>Hadoop looks in <filename>lib/native</filename> for .so
files. HBase looks in
<filename>lib/native/PLATFORM</filename>. See the
<command>bin/hbase</command>.
View the file and look for <varname>native</varname>. See how
we
do the work to find out what platform we are running on
running a little java program
<classname>org.apache.hadoop.util.PlatformName</classname> to
figure it out.
We'll then add <filename>./lib/native/PLATFORM</filename> to
the
<varname>LD_LIBRARY_PATH</varname> environment for when the
JVM starts.
The JVM will look in here (as well as in any other dirs
specified on LD_LIBRARY_PATH)
for codec native libs. If you are unable to figure your
'platform', do:
<programlisting>$ ./bin/hbase
org.apache.hadoop.util.PlatformName</programlisting>.
An example platform would be <varname>Linux-amd64-64</varname>.
</para>
</note>
St.Ack
Re: Snappy compression not working with HBase 0.98.3
Posted by Esteban Gutierrez <es...@cloudera.com>.
Hello Ankit,
The only reason the test can fail in the master is that the snappy natives
libraries are not installed correctly . Have you tried to run the
compression test (hbase org.apache.hadoop.hbase.util.CompressionTest
file:///tmp snappy) in the master? does it works? If it works correctly
then you only need to restart the HBase masters in order to get it working.
cheers,
esteban.
--
Cloudera, Inc.
On Fri, Jul 11, 2014 at 10:11 PM, Ankit Jain <an...@gmail.com>
wrote:
> Hi,
>
> Thanks all for reply..
>
> We have followed the steps mentioned in HBase book. Also, we have set the
> HBASE_LIBRARY_PATH into hbase-env.sh configuration file but still getting
> the above error.
>
> Regards,
> Ankit
>
>
>
>
>
> On Fri, Jul 11, 2014 at 11:11 PM, Esteban Gutierrez <es...@cloudera.com>
> wrote:
>
> > Hello Hanish,
> >
> > Since 0.95 a test for compression was added to the HBase Master, now you
> > need to make sure the native libraries are installed in the HBase
> Master(s)
> > and not just in the Region Servers. (see HBASE-6370 for details about
> this
> > change)
> >
> > Regards,
> > Esteban.
> >
> >
> >
> >
> >
> > --
> > Cloudera, Inc.
> >
> >
> >
> > On Fri, Jul 11, 2014 at 7:14 AM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > Please see
> > > http://hbase.apache.org/book.html#snappy.compression.installation
> > >
> > > Cheers
> > >
> > >
> > > On Fri, Jul 11, 2014 at 3:37 AM, Hanish Bansal <
> > > hanish.bansal.agarwal@gmail.com> wrote:
> > >
> > > > We are using hbase 0.98.3 with hadoop 2.4.0.
> > > >
> > > > Run a compression test using tool, getting below error:
> > > >
> > > > [root@IMPETUS-I0141 hbase-0.98.3-hadoop2]# bin/hbase
> > > > org.apache.hadoop.hbase.util.CompressionTest file:///tmp/test.txt
> > snappy
> > > > 2014-07-11 16:05:10,572 INFO [main] Configuration.deprecation:
> > > > hadoop.native.lib is deprecated. Instead, use io.native.lib.available
> > > > 2014-07-11 16:05:11,006 WARN [main] util.NativeCodeLoader: Unable to
> > > load
> > > > native-hadoop library for your platform... using builtin-java classes
> > > where
> > > > applicable
> > > > 2014-07-11 16:05:11,241 INFO [main] util.ChecksumType: Checksum
> using
> > > > org.apache.hadoop.util.PureJavaCrc32
> > > > 2014-07-11 16:05:11,242 INFO [main] util.ChecksumType: Checksum can
> > use
> > > > org.apache.hadoop.util.PureJavaCrc32C
> > > > Exception in thread "main" java.lang.UnsatisfiedLinkError:
> > > > org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
> > > > at
> > org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native
> > > > Method)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:62)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:131)
> > > > at
> > > >
> > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:147)
> > > > at
> > > >
> > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:162)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:310)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultEncodingContext.<init>(HFileBlockDefaultEncodingContext.java:92)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.<init>(HFileBlock.java:690)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.finishInit(HFileWriterV2.java:117)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.<init>(HFileWriterV2.java:109)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.io.hfile.HFileWriterV2$WriterFactoryV2.createWriter(HFileWriterV2.java:97)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.io.hfile.HFile$WriterFactory.create(HFile.java:393)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.util.CompressionTest.doSmokeTest(CompressionTest.java:118)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.util.CompressionTest.main(CompressionTest.java:148)
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Fri, Jul 11, 2014 at 10:21 AM, Hanish Bansal <
> > > > hanish.bansal.agarwal@gmail.com> wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > Recently i have upgraded HBase environment from 0.94 to 0.98.3. Now
> > > > trying
> > > > > to use snappy compression with it.
> > > > >
> > > > > I have installed snappy library as per guide mentioned in
> > > > > https://hbase.apache.org/book/snappy.compression.html
> > > > >
> > > > > When i am creating a table with snappy compression enabled, i am
> > > getting
> > > > > below error:
> > > > >
> > > > >
> > > > > hbase(main):001:0> create 'test', {NAME=>'cf1',
> > COMPRESSION=>'SNAPPY'}
> > > > > 2014-07-08 20:06:33,265 WARN [main] util.NativeCodeLoader: Unable
> to
> > > > load
> > > > > native-hadoop library for your platform... using builtin-java
> classes
> > > > where
> > > > > applicable
> > > > >
> > > > > ERROR: java.io.IOException: Compression algorithm 'snappy'
> previously
> > > > > failed test.
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:85)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.master.HMaster.checkCompression(HMaster.java:1774)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.master.HMaster.checkCompression(HMaster.java:1767)
> > > > > at
> > > org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1749)
> > > > > at
> > > org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1784)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:40470)
> > > > > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2012)
> > > > > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:73)
> > > > > at
> > > >
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > > > > at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> > > > > at
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > > > at
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > > > at java.lang.Thread.run(Thread.java:722)
> > > > >
> > > > >
> > > > > Please let me know if anyone aware of this issue.
> > > > >
> > > > > --
> > > > > *Thanks & Regards*
> > > > > *Hanish Bansal*
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > *Thanks & Regards*
> > > > *Hanish Bansal*
> > > >
> > >
> >
>
>
>
> --
> Thanks,
> Ankit Jain
>
Re: Snappy compression not working with HBase 0.98.3
Posted by Ankit Jain <an...@gmail.com>.
Hi,
Thanks all for reply..
We have followed the steps mentioned in HBase book. Also, we have set the
HBASE_LIBRARY_PATH into hbase-env.sh configuration file but still getting
the above error.
Regards,
Ankit
On Fri, Jul 11, 2014 at 11:11 PM, Esteban Gutierrez <es...@cloudera.com>
wrote:
> Hello Hanish,
>
> Since 0.95 a test for compression was added to the HBase Master, now you
> need to make sure the native libraries are installed in the HBase Master(s)
> and not just in the Region Servers. (see HBASE-6370 for details about this
> change)
>
> Regards,
> Esteban.
>
>
>
>
>
> --
> Cloudera, Inc.
>
>
>
> On Fri, Jul 11, 2014 at 7:14 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > Please see
> > http://hbase.apache.org/book.html#snappy.compression.installation
> >
> > Cheers
> >
> >
> > On Fri, Jul 11, 2014 at 3:37 AM, Hanish Bansal <
> > hanish.bansal.agarwal@gmail.com> wrote:
> >
> > > We are using hbase 0.98.3 with hadoop 2.4.0.
> > >
> > > Run a compression test using tool, getting below error:
> > >
> > > [root@IMPETUS-I0141 hbase-0.98.3-hadoop2]# bin/hbase
> > > org.apache.hadoop.hbase.util.CompressionTest file:///tmp/test.txt
> snappy
> > > 2014-07-11 16:05:10,572 INFO [main] Configuration.deprecation:
> > > hadoop.native.lib is deprecated. Instead, use io.native.lib.available
> > > 2014-07-11 16:05:11,006 WARN [main] util.NativeCodeLoader: Unable to
> > load
> > > native-hadoop library for your platform... using builtin-java classes
> > where
> > > applicable
> > > 2014-07-11 16:05:11,241 INFO [main] util.ChecksumType: Checksum using
> > > org.apache.hadoop.util.PureJavaCrc32
> > > 2014-07-11 16:05:11,242 INFO [main] util.ChecksumType: Checksum can
> use
> > > org.apache.hadoop.util.PureJavaCrc32C
> > > Exception in thread "main" java.lang.UnsatisfiedLinkError:
> > > org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
> > > at
> org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native
> > > Method)
> > > at
> > >
> > >
> >
> org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:62)
> > > at
> > >
> > >
> >
> org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:131)
> > > at
> > >
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:147)
> > > at
> > >
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:162)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:310)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultEncodingContext.<init>(HFileBlockDefaultEncodingContext.java:92)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.<init>(HFileBlock.java:690)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.finishInit(HFileWriterV2.java:117)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.<init>(HFileWriterV2.java:109)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.io.hfile.HFileWriterV2$WriterFactoryV2.createWriter(HFileWriterV2.java:97)
> > > at
> > >
> >
> org.apache.hadoop.hbase.io.hfile.HFile$WriterFactory.create(HFile.java:393)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.util.CompressionTest.doSmokeTest(CompressionTest.java:118)
> > > at
> > >
> >
> org.apache.hadoop.hbase.util.CompressionTest.main(CompressionTest.java:148)
> > >
> > >
> > >
> > >
> > >
> > > On Fri, Jul 11, 2014 at 10:21 AM, Hanish Bansal <
> > > hanish.bansal.agarwal@gmail.com> wrote:
> > >
> > > > Hi All,
> > > >
> > > > Recently i have upgraded HBase environment from 0.94 to 0.98.3. Now
> > > trying
> > > > to use snappy compression with it.
> > > >
> > > > I have installed snappy library as per guide mentioned in
> > > > https://hbase.apache.org/book/snappy.compression.html
> > > >
> > > > When i am creating a table with snappy compression enabled, i am
> > getting
> > > > below error:
> > > >
> > > >
> > > > hbase(main):001:0> create 'test', {NAME=>'cf1',
> COMPRESSION=>'SNAPPY'}
> > > > 2014-07-08 20:06:33,265 WARN [main] util.NativeCodeLoader: Unable to
> > > load
> > > > native-hadoop library for your platform... using builtin-java classes
> > > where
> > > > applicable
> > > >
> > > > ERROR: java.io.IOException: Compression algorithm 'snappy' previously
> > > > failed test.
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:85)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.master.HMaster.checkCompression(HMaster.java:1774)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.master.HMaster.checkCompression(HMaster.java:1767)
> > > > at
> > org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1749)
> > > > at
> > org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1784)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:40470)
> > > > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2012)
> > > > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:73)
> > > > at
> > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > > > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> > > > at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> > > > at
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > > at
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > > at java.lang.Thread.run(Thread.java:722)
> > > >
> > > >
> > > > Please let me know if anyone aware of this issue.
> > > >
> > > > --
> > > > *Thanks & Regards*
> > > > *Hanish Bansal*
> > > >
> > >
> > >
> > >
> > > --
> > > *Thanks & Regards*
> > > *Hanish Bansal*
> > >
> >
>
--
Thanks,
Ankit Jain
Re: Snappy compression not working with HBase 0.98.3
Posted by Esteban Gutierrez <es...@cloudera.com>.
Hello Hanish,
Since 0.95 a test for compression was added to the HBase Master, now you
need to make sure the native libraries are installed in the HBase Master(s)
and not just in the Region Servers. (see HBASE-6370 for details about this
change)
Regards,
Esteban.
--
Cloudera, Inc.
On Fri, Jul 11, 2014 at 7:14 AM, Ted Yu <yu...@gmail.com> wrote:
> Please see
> http://hbase.apache.org/book.html#snappy.compression.installation
>
> Cheers
>
>
> On Fri, Jul 11, 2014 at 3:37 AM, Hanish Bansal <
> hanish.bansal.agarwal@gmail.com> wrote:
>
> > We are using hbase 0.98.3 with hadoop 2.4.0.
> >
> > Run a compression test using tool, getting below error:
> >
> > [root@IMPETUS-I0141 hbase-0.98.3-hadoop2]# bin/hbase
> > org.apache.hadoop.hbase.util.CompressionTest file:///tmp/test.txt snappy
> > 2014-07-11 16:05:10,572 INFO [main] Configuration.deprecation:
> > hadoop.native.lib is deprecated. Instead, use io.native.lib.available
> > 2014-07-11 16:05:11,006 WARN [main] util.NativeCodeLoader: Unable to
> load
> > native-hadoop library for your platform... using builtin-java classes
> where
> > applicable
> > 2014-07-11 16:05:11,241 INFO [main] util.ChecksumType: Checksum using
> > org.apache.hadoop.util.PureJavaCrc32
> > 2014-07-11 16:05:11,242 INFO [main] util.ChecksumType: Checksum can use
> > org.apache.hadoop.util.PureJavaCrc32C
> > Exception in thread "main" java.lang.UnsatisfiedLinkError:
> > org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
> > at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native
> > Method)
> > at
> >
> >
> org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:62)
> > at
> >
> >
> org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:131)
> > at
> > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:147)
> > at
> > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:162)
> > at
> >
> >
> org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:310)
> > at
> >
> >
> org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultEncodingContext.<init>(HFileBlockDefaultEncodingContext.java:92)
> > at
> >
> >
> org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.<init>(HFileBlock.java:690)
> > at
> >
> >
> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.finishInit(HFileWriterV2.java:117)
> > at
> >
> >
> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.<init>(HFileWriterV2.java:109)
> > at
> >
> >
> org.apache.hadoop.hbase.io.hfile.HFileWriterV2$WriterFactoryV2.createWriter(HFileWriterV2.java:97)
> > at
> >
> org.apache.hadoop.hbase.io.hfile.HFile$WriterFactory.create(HFile.java:393)
> > at
> >
> >
> org.apache.hadoop.hbase.util.CompressionTest.doSmokeTest(CompressionTest.java:118)
> > at
> >
> org.apache.hadoop.hbase.util.CompressionTest.main(CompressionTest.java:148)
> >
> >
> >
> >
> >
> > On Fri, Jul 11, 2014 at 10:21 AM, Hanish Bansal <
> > hanish.bansal.agarwal@gmail.com> wrote:
> >
> > > Hi All,
> > >
> > > Recently i have upgraded HBase environment from 0.94 to 0.98.3. Now
> > trying
> > > to use snappy compression with it.
> > >
> > > I have installed snappy library as per guide mentioned in
> > > https://hbase.apache.org/book/snappy.compression.html
> > >
> > > When i am creating a table with snappy compression enabled, i am
> getting
> > > below error:
> > >
> > >
> > > hbase(main):001:0> create 'test', {NAME=>'cf1', COMPRESSION=>'SNAPPY'}
> > > 2014-07-08 20:06:33,265 WARN [main] util.NativeCodeLoader: Unable to
> > load
> > > native-hadoop library for your platform... using builtin-java classes
> > where
> > > applicable
> > >
> > > ERROR: java.io.IOException: Compression algorithm 'snappy' previously
> > > failed test.
> > > at
> > >
> >
> org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:85)
> > > at
> > >
> >
> org.apache.hadoop.hbase.master.HMaster.checkCompression(HMaster.java:1774)
> > > at
> > >
> >
> org.apache.hadoop.hbase.master.HMaster.checkCompression(HMaster.java:1767)
> > > at
> org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1749)
> > > at
> org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1784)
> > > at
> > >
> >
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:40470)
> > > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2012)
> > > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
> > > at
> > >
> >
> org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:73)
> > > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> > > at
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > at
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > at java.lang.Thread.run(Thread.java:722)
> > >
> > >
> > > Please let me know if anyone aware of this issue.
> > >
> > > --
> > > *Thanks & Regards*
> > > *Hanish Bansal*
> > >
> >
> >
> >
> > --
> > *Thanks & Regards*
> > *Hanish Bansal*
> >
>
Re: Snappy compression not working with HBase 0.98.3
Posted by Ted Yu <yu...@gmail.com>.
Please see http://hbase.apache.org/book.html#snappy.compression.installation
Cheers
On Fri, Jul 11, 2014 at 3:37 AM, Hanish Bansal <
hanish.bansal.agarwal@gmail.com> wrote:
> We are using hbase 0.98.3 with hadoop 2.4.0.
>
> Run a compression test using tool, getting below error:
>
> [root@IMPETUS-I0141 hbase-0.98.3-hadoop2]# bin/hbase
> org.apache.hadoop.hbase.util.CompressionTest file:///tmp/test.txt snappy
> 2014-07-11 16:05:10,572 INFO [main] Configuration.deprecation:
> hadoop.native.lib is deprecated. Instead, use io.native.lib.available
> 2014-07-11 16:05:11,006 WARN [main] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> 2014-07-11 16:05:11,241 INFO [main] util.ChecksumType: Checksum using
> org.apache.hadoop.util.PureJavaCrc32
> 2014-07-11 16:05:11,242 INFO [main] util.ChecksumType: Checksum can use
> org.apache.hadoop.util.PureJavaCrc32C
> Exception in thread "main" java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
> at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native
> Method)
> at
>
> org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:62)
> at
>
> org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:131)
> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:147)
> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:162)
> at
>
> org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:310)
> at
>
> org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultEncodingContext.<init>(HFileBlockDefaultEncodingContext.java:92)
> at
>
> org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.<init>(HFileBlock.java:690)
> at
>
> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.finishInit(HFileWriterV2.java:117)
> at
>
> org.apache.hadoop.hbase.io.hfile.HFileWriterV2.<init>(HFileWriterV2.java:109)
> at
>
> org.apache.hadoop.hbase.io.hfile.HFileWriterV2$WriterFactoryV2.createWriter(HFileWriterV2.java:97)
> at
> org.apache.hadoop.hbase.io.hfile.HFile$WriterFactory.create(HFile.java:393)
> at
>
> org.apache.hadoop.hbase.util.CompressionTest.doSmokeTest(CompressionTest.java:118)
> at
> org.apache.hadoop.hbase.util.CompressionTest.main(CompressionTest.java:148)
>
>
>
>
>
> On Fri, Jul 11, 2014 at 10:21 AM, Hanish Bansal <
> hanish.bansal.agarwal@gmail.com> wrote:
>
> > Hi All,
> >
> > Recently i have upgraded HBase environment from 0.94 to 0.98.3. Now
> trying
> > to use snappy compression with it.
> >
> > I have installed snappy library as per guide mentioned in
> > https://hbase.apache.org/book/snappy.compression.html
> >
> > When i am creating a table with snappy compression enabled, i am getting
> > below error:
> >
> >
> > hbase(main):001:0> create 'test', {NAME=>'cf1', COMPRESSION=>'SNAPPY'}
> > 2014-07-08 20:06:33,265 WARN [main] util.NativeCodeLoader: Unable to
> load
> > native-hadoop library for your platform... using builtin-java classes
> where
> > applicable
> >
> > ERROR: java.io.IOException: Compression algorithm 'snappy' previously
> > failed test.
> > at
> >
> org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:85)
> > at
> >
> org.apache.hadoop.hbase.master.HMaster.checkCompression(HMaster.java:1774)
> > at
> >
> org.apache.hadoop.hbase.master.HMaster.checkCompression(HMaster.java:1767)
> > at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1749)
> > at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1784)
> > at
> >
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:40470)
> > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2012)
> > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
> > at
> >
> org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:73)
> > at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:722)
> >
> >
> > Please let me know if anyone aware of this issue.
> >
> > --
> > *Thanks & Regards*
> > *Hanish Bansal*
> >
>
>
>
> --
> *Thanks & Regards*
> *Hanish Bansal*
>
Re: Snappy compression not working with HBase 0.98.3
Posted by Hanish Bansal <ha...@gmail.com>.
We are using hbase 0.98.3 with hadoop 2.4.0.
Run a compression test using tool, getting below error:
[root@IMPETUS-I0141 hbase-0.98.3-hadoop2]# bin/hbase
org.apache.hadoop.hbase.util.CompressionTest file:///tmp/test.txt snappy
2014-07-11 16:05:10,572 INFO [main] Configuration.deprecation:
hadoop.native.lib is deprecated. Instead, use io.native.lib.available
2014-07-11 16:05:11,006 WARN [main] util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
2014-07-11 16:05:11,241 INFO [main] util.ChecksumType: Checksum using
org.apache.hadoop.util.PureJavaCrc32
2014-07-11 16:05:11,242 INFO [main] util.ChecksumType: Checksum can use
org.apache.hadoop.util.PureJavaCrc32C
Exception in thread "main" java.lang.UnsatisfiedLinkError:
org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native
Method)
at
org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:62)
at
org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:131)
at
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:147)
at
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:162)
at
org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:310)
at
org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultEncodingContext.<init>(HFileBlockDefaultEncodingContext.java:92)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.<init>(HFileBlock.java:690)
at
org.apache.hadoop.hbase.io.hfile.HFileWriterV2.finishInit(HFileWriterV2.java:117)
at
org.apache.hadoop.hbase.io.hfile.HFileWriterV2.<init>(HFileWriterV2.java:109)
at
org.apache.hadoop.hbase.io.hfile.HFileWriterV2$WriterFactoryV2.createWriter(HFileWriterV2.java:97)
at
org.apache.hadoop.hbase.io.hfile.HFile$WriterFactory.create(HFile.java:393)
at
org.apache.hadoop.hbase.util.CompressionTest.doSmokeTest(CompressionTest.java:118)
at
org.apache.hadoop.hbase.util.CompressionTest.main(CompressionTest.java:148)
On Fri, Jul 11, 2014 at 10:21 AM, Hanish Bansal <
hanish.bansal.agarwal@gmail.com> wrote:
> Hi All,
>
> Recently i have upgraded HBase environment from 0.94 to 0.98.3. Now trying
> to use snappy compression with it.
>
> I have installed snappy library as per guide mentioned in
> https://hbase.apache.org/book/snappy.compression.html
>
> When i am creating a table with snappy compression enabled, i am getting
> below error:
>
>
> hbase(main):001:0> create 'test', {NAME=>'cf1', COMPRESSION=>'SNAPPY'}
> 2014-07-08 20:06:33,265 WARN [main] util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
>
> ERROR: java.io.IOException: Compression algorithm 'snappy' previously
> failed test.
> at
> org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:85)
> at
> org.apache.hadoop.hbase.master.HMaster.checkCompression(HMaster.java:1774)
> at
> org.apache.hadoop.hbase.master.HMaster.checkCompression(HMaster.java:1767)
> at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1749)
> at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1784)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:40470)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2012)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
> at
> org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:73)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
>
>
> Please let me know if anyone aware of this issue.
>
> --
> *Thanks & Regards*
> *Hanish Bansal*
>
--
*Thanks & Regards*
*Hanish Bansal*