You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@hbase.apache.org by elsif <el...@gmail.com> on 2009/09/29 22:48:46 UTC

Compression test utility

Attached is a stand alone utility to test compression compatibility
before starting hbase:

# ./hbase org.apache.hadoop.hbase.test.HFileCompressionTest
Usage: HFileCompressionTest hdfs compression
  compression must be one of [ none, gz, lzo ]

# ./hbase org.apache.hadoop.hbase.test.HFileCompressionTest
hdfs://localhost:8020/ lzo
09/09/29 13:39:55 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
09/09/29 13:39:55 INFO lzo.LzoCodec: Successfully loaded & initialized
native-lzo library
09/09/29 13:39:55 INFO compress.CodecPool: Got brand-new compressor
OK

The utility creates a temporary HFile named '.hfile-comp-test' at the
specified HDFS path and outputs either 'OK' or 'FAILED'.

Re: Compression test utility

Posted by elsif <el...@gmail.com>.

The code has been attached to HBASE-1875.

Regards,
Jason


stack wrote:
> Excellent.
>
> Please stick your code up in an issue.  Should we add documentation on this
> utility to Ryan's LZO page up on hbase.org wiki?
>
> Good stuff,
> St.Ack
>
>

Re: Compression test utility

Posted by stack <st...@duboce.net>.

Excellent.

Please stick your code up in an issue.  Should we add documentation on this
utility to Ryan's LZO page up on hbase.org wiki?

Good stuff,
St.Ack

On Tue, Sep 29, 2009 at 1:48 PM, elsif <el...@gmail.com> wrote:

> Attached is a stand alone utility to test compression compatibility
> before starting hbase:
>
> # ./hbase org.apache.hadoop.hbase.test.HFileCompressionTest
> Usage: HFileCompressionTest hdfs compression
>  compression must be one of [ none, gz, lzo ]
>
> # ./hbase org.apache.hadoop.hbase.test.HFileCompressionTest
> hdfs://localhost:8020/ lzo
> 09/09/29 13:39:55 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
> 09/09/29 13:39:55 INFO lzo.LzoCodec: Successfully loaded & initialized
> native-lzo library
> 09/09/29 13:39:55 INFO compress.CodecPool: Got brand-new compressor
> OK
>
> The utility creates a temporary HFile named '.hfile-comp-test' at the
> specified HDFS path and outputs either 'OK' or 'FAILED'.
>
>
>

Re: Compression test utility

Posted by Ryan Rawson <ry...@gmail.com>.

Thanks!

Don't forget, that this has to be run on every single node.

I have personally found that once you get the native libs and the JAR
in place and rsynced across, things are peachy.

Good luck!
-ryan

On Tue, Sep 29, 2009 at 1:48 PM, elsif <el...@gmail.com> wrote:
> Attached is a stand alone utility to test compression compatibility
> before starting hbase:
>
> # ./hbase org.apache.hadoop.hbase.test.HFileCompressionTest
> Usage: HFileCompressionTest hdfs compression
>  compression must be one of [ none, gz, lzo ]
>
> # ./hbase org.apache.hadoop.hbase.test.HFileCompressionTest
> hdfs://localhost:8020/ lzo
> 09/09/29 13:39:55 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
> 09/09/29 13:39:55 INFO lzo.LzoCodec: Successfully loaded & initialized
> native-lzo library
> 09/09/29 13:39:55 INFO compress.CodecPool: Got brand-new compressor
> OK
>
> The utility creates a temporary HFile named '.hfile-comp-test' at the
> specified HDFS path and outputs either 'OK' or 'FAILED'.
>
>
>

Re: Compression test utility

Posted by stephen mulcahy <st...@deri.org>.

elsif wrote:
> Attached is a stand alone utility to test compression compatibility
> before starting hbase:
> 
> # ./hbase org.apache.hadoop.hbase.test.HFileCompressionTest
> Usage: HFileCompressionTest hdfs compression
>   compression must be one of [ none, gz, lzo ]

Hi,

Thanks for this. I found it very useful this morning. Can we get this 
included in the standard distribution?

For those not so familiar with building and running code against hbase, 
the following detailed steps may be useful,

1. Download compression test util from
http://mail-archives.apache.org/mod_mbox/hadoop-hbase-user/200909.mbox/%3C4AC272
AE.3080105@gmail.com%3E

2. Compile

(change .jar's below to match your hadoop and hbase versions)

javac -cp hadoop/hadoop-0.20.1-core.jar:hbase/hbase-0.20.0.jar 
HFileCompressionTest.java

3. "Install"
mkdir -p build/org/apache/hadoop/hbase/test
mv HFileCompressionTest.class build/org/apache/hadoop/hbase/test

4. Run
export HBASE_CLASSPATH=build
hbase/bin/hbase org.apache.hadoop.hbase.test.HFileCompressionTest 
hdfs://hbase.example.org:9000/ lzo

-stephen

-- 
Stephen Mulcahy, DI2, Digital Enterprise Research Institute,
NUI Galway, IDA Business Park, Lower Dangan, Galway, Ireland
http://di2.deri.ie    http://webstar.deri.ie    http://sindice.com