You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Alex Soto <al...@envieta.com> on 2018/05/01 13:35:12 UTC
Snappy compression with Pig
Hello,
I am using Pig 0.17.0 and I am trying to enable Snappy compression for temporary files.
I installed Snappy on all the Hadoop nodes:
sudo yum install snappy snappy-devel
ln -sf /usr/lib64/libsnappy.so /opt/hadoop/lib/native/
Yum installed the following packages:
Installed Packages
Name : snappy
Arch : x86_64
Version : 1.1.0
Release : 3.el7
Size : 117 k
Repo : installed
From repo : anaconda
Summary : Fast compression and decompression library
URL : http://code.google.com/p/snappy/
License : BSD
Description : Snappy is a compression/decompression library. It does not aim for maximum
: compression, or compatibility with any other compression library; instead, it
: aims for very high speeds and reasonable compression. For instance, compared to
: the fastest mode of zlib, Snappy is an order of magnitude faster for most
: inputs, but the resulting compressed files are anywhere from 20% to 100%
: bigger.
Name : snappy-devel
Arch : x86_64
Version : 1.1.0
Release : 3.el7
Size : 29 k
Repo : installed
From repo : base
Summary : Development files for snappy
URL : http://code.google.com/p/snappy/
License : BSD
Description : The snappy-devel package contains libraries and header files for
: developing applications that use snappy.
In the pig.properties file, I made the following changes:
pig.tmpfilecompression=true
pig.tmpfilecompression.storage=seqfile
pig.tmpfilecompression.codec=snappy
Now, when I run my script, I get this error:
java.lang.ClassCastException: class org.xerial.snappy.SnappyCodec
In my Pig’s lib directory I can see this jar:
snappy-java-1.1.1.3.jar
Looks like incompatible version, but I can’t find any information regarding which version of Snappy is compatible with this version of Pig.
I will appreciate any input on this.
Best regards,
Alex soto
Re: Snappy compression with Pig
Posted by Rohini Palaniswamy <ro...@apache.org>.
Can you give the full stack trace?
On Tue, May 1, 2018 at 6:35 AM, Alex Soto <al...@envieta.com> wrote:
> Hello,
>
> I am using Pig 0.17.0 and I am trying to enable Snappy compression for
> temporary files.
> I installed Snappy on all the Hadoop nodes:
>
> sudo yum install snappy snappy-devel
> ln -sf /usr/lib64/libsnappy.so /opt/hadoop/lib/native/
>
> Yum installed the following packages:
>
> Installed Packages
> Name : snappy
> Arch : x86_64
> Version : 1.1.0
> Release : 3.el7
> Size : 117 k
> Repo : installed
> From repo : anaconda
> Summary : Fast compression and decompression library
> URL : http://code.google.com/p/snappy/
> License : BSD
> Description : Snappy is a compression/decompression library. It does not
> aim for maximum
> : compression, or compatibility with any other compression
> library; instead, it
> : aims for very high speeds and reasonable compression. For
> instance, compared to
> : the fastest mode of zlib, Snappy is an order of magnitude
> faster for most
> : inputs, but the resulting compressed files are anywhere from
> 20% to 100%
> : bigger.
>
> Name : snappy-devel
> Arch : x86_64
> Version : 1.1.0
> Release : 3.el7
> Size : 29 k
> Repo : installed
> From repo : base
> Summary : Development files for snappy
> URL : http://code.google.com/p/snappy/
> License : BSD
> Description : The snappy-devel package contains libraries and header files
> for
> : developing applications that use snappy.
>
> In the pig.properties file, I made the following changes:
>
> pig.tmpfilecompression=true
> pig.tmpfilecompression.storage=seqfile
> pig.tmpfilecompression.codec=snappy
>
> Now, when I run my script, I get this error:
>
>
> java.lang.ClassCastException: class org.xerial.snappy.SnappyCodec
>
>
> In my Pig’s lib directory I can see this jar:
>
> snappy-java-1.1.1.3.jar
>
> Looks like incompatible version, but I can’t find any information
> regarding which version of Snappy is compatible with this version of Pig.
> I will appreciate any input on this.
>
> Best regards,
> Alex soto
>
>
>
>