You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by schausson <sc...@softera.fr> on 2017/10/03 17:21:47 UTC
Using snappy compression with standalone HBase ?
Hi,
I'm experimenting with HBase on a brand new linux VM (ubuntu), as a
standalone installation (I don't have any hadoop distribution on my VM, it's
worth saying it). I would like to test compression options, but couldn't
figure out how to make it working :
I manually installed snappy stuff (apt-get install libsnappy1v5
libsnappy-dev), so the library is there, but how to point to it ?
I figured out that following command "hbase --config ~/conf_hbase
org.apache.hadoop.util.NativeLibraryChecker" exclusively relies on hadoop
and doesn't display compression libraries even if they have been installed.
I set HBASE_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/ (where snappy .so files
are located) and gave a try to "hbase
org.apache.hadoop.hbase.util.CompressionTest
/path/to/my/file/on/local/filesystem snappy", but faced following exception
:
Exception in thread "main" java.lang.UnsatisfiedLinkError:
org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native
Method)
at
org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63)
at
org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:132)
at
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
Did anybody already make this work together ?
Thanks for your help
--
Sent from: http://apache-hbase.679495.n3.nabble.com/HBase-User-f4020416.html
Re: Using snappy compression with standalone HBase ?
Posted by schausson <sc...@softera.fr>.
Hi,
Finally I figured out : I had to download libhadoop.so and reference its
location with HBASE_LIBRARY_PATH
Now it works fine !
--
Sent from: http://apache-hbase.679495.n3.nabble.com/HBase-User-f4020416.html
Re: Using snappy compression with standalone HBase ?
Posted by schausson <sc...@softera.fr>.
Hi Ted,
Thanks for your help,
I deployed Hbase 1.2.5, and in the lib folder, I can see a bunch of hadoop
jars, all of them for the 2.5.1 release :
hadoop-annotations-2.5.1.jar
hadoop-auth-2.5.1.jar
hadoop-client-2.5.1.jar
hadoop-common-2.5.1.jar
hadoop-hdfs-2.5.1.jar
hadoop-mapreduce-client-app-2.5.1.jar
hadoop-mapreduce-client-common-2.5.1.jar
hadoop-mapreduce-client-core-2.5.1.jar
hadoop-mapreduce-client-jobclient-2.5.1.jar
hadoop-mapreduce-client-shuffle-2.5.1.jar
hadoop-yarn-api-2.5.1.jar
hadoop-yarn-client-2.5.1.jar
hadoop-yarn-common-2.5.1.jar
hadoop-yarn-server-common-2.5.1.jar
Is there something missing ?
Thanks again
--
Sent from: http://apache-hbase.679495.n3.nabble.com/HBase-User-f4020416.html
Re: Using snappy compression with standalone HBase ?
Posted by Ted Yu <yu...@gmail.com>.
bq. org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
The method is defined
in hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/NativeCodeLoader.java
Which release of hbase did you deploy ?
Which hadoop jars are under lib dir ?
Thanks
On Tue, Oct 3, 2017 at 10:21 AM, schausson <sc...@softera.fr> wrote:
> Hi,
> I'm experimenting with HBase on a brand new linux VM (ubuntu), as a
> standalone installation (I don't have any hadoop distribution on my VM,
> it's
> worth saying it). I would like to test compression options, but couldn't
> figure out how to make it working :
> I manually installed snappy stuff (apt-get install libsnappy1v5
> libsnappy-dev), so the library is there, but how to point to it ?
>
> I figured out that following command "hbase --config ~/conf_hbase
> org.apache.hadoop.util.NativeLibraryChecker" exclusively relies on hadoop
> and doesn't display compression libraries even if they have been installed.
>
> I set HBASE_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/ (where snappy .so
> files
> are located) and gave a try to "hbase
> org.apache.hadoop.hbase.util.CompressionTest
> /path/to/my/file/on/local/filesystem snappy", but faced following
> exception
> :
> Exception in thread "main" java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
> at org.apache.hadoop.util.NativeCodeLoader.
> buildSupportsSnappy(Native
> Method)
> at
> org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(
> SnappyCodec.java:63)
> at
> org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(
> SnappyCodec.java:132)
> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
>
> Did anybody already make this work together ?
> Thanks for your help
>
>
>
>
> --
> Sent from: http://apache-hbase.679495.n3.nabble.com/HBase-User-
> f4020416.html
>