You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by Massimilian Mattetti <MA...@il.ibm.com> on 2017/02/07 11:18:01 UTC
how to make LZO compression work?
Hi all,
I got stuck trying to enable the LZO compression on a table. I installed
the native-lzo library on each tablet server (sudo apt-get install
liblzo2-dev) and set the compression codecs libraries in the Hadoop
core-site.xml:
<property>
<name>io.compression.codecs</name>
<value>org.apache.hadoop.io.compress.GzipCodec,
org.apache.hadoop.io.compress.DefaultCodec,
org.apache.hadoop.io.compress.BZip2Codec,
com.hadoop.compression.lzo.LzoCodec,
com.hadoop.compression.lzo.LzopCodec
</value>
</property>
<property>
<name>io.compression.codec.lzo.class</name>
<value>com.hadoop.compression.lzo.LzoCodec</value>
</property>
I also added the path to the library to the general.classpaths property
of accumulo-site.xml:
<property>
<name>general.classpaths</name>
<value>
....
/usr/hdp/current/hadoop-client/lib/hadoop-lzo-0.6.0.2.5.0.0-1245.jar,
/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64/libgplcompression.so,
/lib/x86_64-linux-gnu/liblzo2.so.2
</value>
</property>
I tested that the lzo library is working correctly compressing a file via
a MapReduce Job, but when I asked Accumulo to compact the table on which I
enabled the LZO compression I got this error on each tablet server:
MajC Unexpected exception, extent = 1z;???????????<;???????????.
java.lang.RuntimeException: native-lzo library not available
at
com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.java:155)
at
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)
at
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:165)
at
org.apache.accumulo.core.file.rfile.bcfile.Compression$Algorithm.getCompressor(Compression.java:502)
at
org.apache.accumulo.core.file.rfile.bcfile.BCFile$Writer$WBlockState.<init>(BCFile.java:192)
at
org.apache.accumulo.core.file.rfile.bcfile.BCFile$Writer.prepareDataBlock(BCFile.java:487)
at
org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Writer.prepareDataBlock(CachableBlockFile.java:89)
at
org.apache.accumulo.core.file.rfile.RFile$LocalityGroupWriter.append(RFile.java:429)
at
org.apache.accumulo.core.file.rfile.RFile$Writer.append(RFile.java:596)
at
org.apache.accumulo.core.file.BloomFilterLayer$Writer.append(BloomFilterLayer.java:141)
at
org.apache.accumulo.tserver.tablet.Compactor.compactLocalityGroup(Compactor.java:367)
at
org.apache.accumulo.tserver.tablet.Compactor.call(Compactor.java:223)
at
org.apache.accumulo.tserver.tablet.Tablet._majorCompact(Tablet.java:1850)
at
org.apache.accumulo.tserver.tablet.Tablet.majorCompact(Tablet.java:1967)
at
org.apache.accumulo.tserver.tablet.CompactionRunner.run(CompactionRunner.java:44)
at
org.apache.htrace.wrappers.TraceRunnable.run(TraceRunnable.java:57)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
at java.lang.Thread.run(Thread.java:745)
What am I missing?
Thanks
Regards,
Max
Re: how to make LZO compression work?
Posted by Massimilian Mattetti <MA...@il.ibm.com>.
I did the two steps you suggested and it worked.
Thanks.
Regards,
Max
From: Sean Busbey <bu...@cloudera.com>
To: Accumulo User List <us...@accumulo.apache.org>
Date: 07/02/2017 18:00
Subject: Re: how to make LZO compression work?
The documentation put out by Cloudera for running Accumulo on CDH used
to have a section on getting LZO working[1].
One thing I see in those docs that I don't see you doing an equivalent
of is setting LD_LIBRARY_PATH to point to the LZO libraries. They also
specify io.compression.codec.lzo.class as a command line argument for
the service startup rather than putting it in core-site.xml, but it's
not clear to me if that's necessary.
[1]:
http://www.cloudera.com/documentation/other/accumulo/1-6-0/PDF/Apache-Accumulo-Installation-Guide.pdf
On Tue, Feb 7, 2017 at 5:18 AM, Massimilian Mattetti
<MA...@il.ibm.com> wrote:
> Hi all,
>
> I got stuck trying to enable the LZO compression on a table. I installed
the
> native-lzo library on each tablet server (sudo apt-get install
liblzo2-dev)
> and set the compression codecs libraries in the Hadoop core-site.xml:
> <property>
> <name>io.compression.codecs</name>
> <value>org.apache.hadoop.io.compress.GzipCodec,
> org.apache.hadoop.io.compress.DefaultCodec,
> org.apache.hadoop.io.compress.BZip2Codec,
> com.hadoop.compression.lzo.LzoCodec,
> com.hadoop.compression.lzo.LzopCodec
> </value>
> </property>
> <property>
> <name>io.compression.codec.lzo.class</name>
> <value>com.hadoop.compression.lzo.LzoCodec</value>
> </property>
>
> I also added the path to the library to the general.classpaths property
of
> accumulo-site.xml:
> <property>
> <name>general.classpaths</name>
> <value>
> ....
>
> /usr/hdp/current/hadoop-client/lib/hadoop-lzo-0.6.0.2.5.0.0-1245.jar,
>
>
/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64/libgplcompression.so,
> /lib/x86_64-linux-gnu/liblzo2.so.2
> </value>
> </property>
>
> I tested that the lzo library is working correctly compressing a file
via a
> MapReduce Job, but when I asked Accumulo to compact the table on which I
> enabled the LZO compression I got this error on each tablet server:
> MajC Unexpected exception, extent = 1z; ? ??????????<; ? ??????????.
> java.lang.RuntimeException: native-lzo library not available
> at
> com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.java:155)
> at
>
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)
> at
>
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:165)
> at
>
org.apache.accumulo.core.file.rfile.bcfile.Compression$Algorithm.getCompressor(Compression.java:502)
> at
>
org.apache.accumulo.core.file.rfile.bcfile.BCFile$Writer$WBlockState.<init>(BCFile.java:192)
> at
>
org.apache.accumulo.core.file.rfile.bcfile.BCFile$Writer.prepareDataBlock(BCFile.java:487)
> at
>
org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Writer.prepareDataBlock(CachableBlockFile.java:89)
> at
>
org.apache.accumulo.core.file.rfile.RFile$LocalityGroupWriter.append(RFile.java:429)
> at
> org.apache.accumulo.core.file.rfile.RFile$Writer.append(RFile.java:596)
> at
>
org.apache.accumulo.core.file.BloomFilterLayer$Writer.append(BloomFilterLayer.java:141)
> at
>
org.apache.accumulo.tserver.tablet.Compactor.compactLocalityGroup(Compactor.java:367)
> at
> org.apache.accumulo.tserver.tablet.Compactor.call(Compactor.java:223)
> at
>
org.apache.accumulo.tserver.tablet.Tablet._majorCompact(Tablet.java:1850)
> at
> org.apache.accumulo.tserver.tablet.Tablet.majorCompact(Tablet.java:1967)
> at
>
org.apache.accumulo.tserver.tablet.CompactionRunner.run(CompactionRunner.java:44)
> at
> org.apache.htrace.wrappers.TraceRunnable.run(TraceRunnable.java:57)
> at
>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at
>
org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
> at java.lang.Thread.run(Thread.java:745)
> What am I missing?
> Thanks
>
> Regards,
> Max
>
--
busbey
Re: how to make LZO compression work?
Posted by Sean Busbey <bu...@cloudera.com>.
The documentation put out by Cloudera for running Accumulo on CDH used
to have a section on getting LZO working[1].
One thing I see in those docs that I don't see you doing an equivalent
of is setting LD_LIBRARY_PATH to point to the LZO libraries. They also
specify io.compression.codec.lzo.class as a command line argument for
the service startup rather than putting it in core-site.xml, but it's
not clear to me if that's necessary.
[1]: http://www.cloudera.com/documentation/other/accumulo/1-6-0/PDF/Apache-Accumulo-Installation-Guide.pdf
On Tue, Feb 7, 2017 at 5:18 AM, Massimilian Mattetti
<MA...@il.ibm.com> wrote:
> Hi all,
>
> I got stuck trying to enable the LZO compression on a table. I installed the
> native-lzo library on each tablet server (sudo apt-get install liblzo2-dev)
> and set the compression codecs libraries in the Hadoop core-site.xml:
> <property>
> <name>io.compression.codecs</name>
> <value>org.apache.hadoop.io.compress.GzipCodec,
> org.apache.hadoop.io.compress.DefaultCodec,
> org.apache.hadoop.io.compress.BZip2Codec,
> com.hadoop.compression.lzo.LzoCodec,
> com.hadoop.compression.lzo.LzopCodec
> </value>
> </property>
> <property>
> <name>io.compression.codec.lzo.class</name>
> <value>com.hadoop.compression.lzo.LzoCodec</value>
> </property>
>
> I also added the path to the library to the general.classpaths property of
> accumulo-site.xml:
> <property>
> <name>general.classpaths</name>
> <value>
> ....
>
> /usr/hdp/current/hadoop-client/lib/hadoop-lzo-0.6.0.2.5.0.0-1245.jar,
>
> /usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64/libgplcompression.so,
> /lib/x86_64-linux-gnu/liblzo2.so.2
> </value>
> </property>
>
> I tested that the lzo library is working correctly compressing a file via a
> MapReduce Job, but when I asked Accumulo to compact the table on which I
> enabled the LZO compression I got this error on each tablet server:
> MajC Unexpected exception, extent = 1z; ? ??????????<; ? ??????????.
> java.lang.RuntimeException: native-lzo library not available
> at
> com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.java:155)
> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)
> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:165)
> at
> org.apache.accumulo.core.file.rfile.bcfile.Compression$Algorithm.getCompressor(Compression.java:502)
> at
> org.apache.accumulo.core.file.rfile.bcfile.BCFile$Writer$WBlockState.<init>(BCFile.java:192)
> at
> org.apache.accumulo.core.file.rfile.bcfile.BCFile$Writer.prepareDataBlock(BCFile.java:487)
> at
> org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Writer.prepareDataBlock(CachableBlockFile.java:89)
> at
> org.apache.accumulo.core.file.rfile.RFile$LocalityGroupWriter.append(RFile.java:429)
> at
> org.apache.accumulo.core.file.rfile.RFile$Writer.append(RFile.java:596)
> at
> org.apache.accumulo.core.file.BloomFilterLayer$Writer.append(BloomFilterLayer.java:141)
> at
> org.apache.accumulo.tserver.tablet.Compactor.compactLocalityGroup(Compactor.java:367)
> at
> org.apache.accumulo.tserver.tablet.Compactor.call(Compactor.java:223)
> at
> org.apache.accumulo.tserver.tablet.Tablet._majorCompact(Tablet.java:1850)
> at
> org.apache.accumulo.tserver.tablet.Tablet.majorCompact(Tablet.java:1967)
> at
> org.apache.accumulo.tserver.tablet.CompactionRunner.run(CompactionRunner.java:44)
> at
> org.apache.htrace.wrappers.TraceRunnable.run(TraceRunnable.java:57)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
> at java.lang.Thread.run(Thread.java:745)
> What am I missing?
> Thanks
>
> Regards,
> Max
>
--
busbey
Re: how to make LZO compression work?
Posted by Josh Elser <jo...@gmail.com>.
If you're amenable to it, Snappy should have similar performance without
the need for the native library :)
Massimilian Mattetti wrote:
> Hi all,
>
> I got stuck trying to enable the LZO compression on a table. I installed
> the native-lzo library on each tablet server (sudo apt-get install
> liblzo2-dev) and set the compression codecs libraries in the Hadoop
> core-site.xml:
> <property>
> <name>io.compression.codecs</name>
> <value>org.apache.hadoop.io.compress.GzipCodec,
> org.apache.hadoop.io.compress.DefaultCodec,
> org.apache.hadoop.io.compress.BZip2Codec,
> com.hadoop.compression.lzo.LzoCodec,
> com.hadoop.compression.lzo.LzopCodec
> </value>
> </property>
> <property>
> <name>io.compression.codec.lzo.class</name>
> <value>com.hadoop.compression.lzo.LzoCodec</value>
> </property>
>
> I also added the path to the library to the general.classpaths property
> of accumulo-site.xml:
> <property>
> <name>general.classpaths</name>
> <value>
> ....
> /usr/hdp/current/hadoop-client/lib/hadoop-lzo-0.6.0.2.5.0.0-1245.jar,
> /usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64/libgplcompression.so,
> /lib/x86_64-linux-gnu/liblzo2.so.2
> </value>
> </property>
>
> I tested that the lzo library is working correctly compressing a file
> via a MapReduce Job, but when I asked Accumulo to compact the table on
> which I enabled the LZO compression I got this error on each tablet server:
> MajC Unexpected exception, extent = 1z;???????????<;???????????.
> java.lang.RuntimeException: native-lzo library not available
> at com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.java:155)
> at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)
> at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:165)
> at
> org.apache.accumulo.core.file.rfile.bcfile.Compression$Algorithm.getCompressor(Compression.java:502)
> at
> org.apache.accumulo.core.file.rfile.bcfile.BCFile$Writer$WBlockState.<init>(BCFile.java:192)
> at
> org.apache.accumulo.core.file.rfile.bcfile.BCFile$Writer.prepareDataBlock(BCFile.java:487)
> at
> org.apache.accumulo.core.file.blockfile.impl.CachableBlockFile$Writer.prepareDataBlock(CachableBlockFile.java:89)
> at
> org.apache.accumulo.core.file.rfile.RFile$LocalityGroupWriter.append(RFile.java:429)
> at org.apache.accumulo.core.file.rfile.RFile$Writer.append(RFile.java:596)
> at
> org.apache.accumulo.core.file.BloomFilterLayer$Writer.append(BloomFilterLayer.java:141)
> at
> org.apache.accumulo.tserver.tablet.Compactor.compactLocalityGroup(Compactor.java:367)
> at org.apache.accumulo.tserver.tablet.Compactor.call(Compactor.java:223)
> at org.apache.accumulo.tserver.tablet.Tablet._majorCompact(Tablet.java:1850)
> at org.apache.accumulo.tserver.tablet.Tablet.majorCompact(Tablet.java:1967)
> at
> org.apache.accumulo.tserver.tablet.CompactionRunner.run(CompactionRunner.java:44)
> at org.apache.htrace.wrappers.TraceRunnable.run(TraceRunnable.java:57)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at
> org.apache.accumulo.fate.util.LoggingRunnable.run(LoggingRunnable.java:35)
> at java.lang.Thread.run(Thread.java:745)
> What am I missing?
> Thanks
>
> Regards,
> Max
>