You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by al...@aim.com on 2012/11/06 02:10:10 UTC
problem with hadoop-snappy
Hello,
I use hadoop-1.0.4 I have followed instruction to install hadoop-snappy at
http://code.google.com/p/hadoop-snappy/
When I run a mapred job I see
FATAL org.apache.hadoop.mapred.TaskTracker: Task: attempt_201211051656_0002_m_000000_3 - Killed : org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
in task tracker log and the job fails.
Any ideas how to fix this issue.
Thanks in advance.
Alex.
Re: problem with hadoop-snappy
Posted by Colin McCabe <cm...@alumni.cmu.edu>.
I think what you're hitting is probably
https://issues.apache.org/jira/browse/HADOOP-8756. It was fixed in
branch-2 and branch-3, but not in the old releases.
If you want to work around the problem, try explicitly setting
LD_LIBRARY_PATH to include the directory that contains snappy.
cheers,
Colin
On Tue, Nov 6, 2012 at 1:30 PM, <al...@aim.com> wrote:
> When I used hadoop-1.0.4 without the installation of hadoop-snappy I saw in
> the logs
>
> WARN snappy.LoadSnappy: Snappy native library not loaded
>
> So, it was my understanding that snappy is not included with hadoop-1.0.4.
> After removing the libs I installed, I see again the same warning.
>
> Any ideas how to fix this issue?
>
> Thanks.
> Alex.
>
>
>
>
> -----Original Message-----
> From: Binglin Chang <de...@gmail.com>
> To: user <us...@hadoop.apache.org>
> Sent: Mon, Nov 5, 2012 6:07 pm
> Subject: Re: problem with hadoop-snappy
>
> I think hadoop-1.0.4 already have snappy included, you should not using
> other third party libraries.
>
>
>
> On Tue, Nov 6, 2012 at 9:10 AM, <al...@aim.com> wrote:
>>
>> Hello,
>>
>> I use hadoop-1.0.4 I have followed instruction to install hadoop-snappy
>> at
>> http://code.google.com/p/hadoop-snappy/
>>
>> When I run a mapred job I see
>>
>> FATAL org.apache.hadoop.mapred.TaskTracker: Task:
>> attempt_201211051656_0002_m_000000_3 - Killed :
>> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
>>
>> in task tracker log and the job fails.
>>
>> Any ideas how to fix this issue.
>>
>> Thanks in advance.
>> Alex.
>>
>
Re: problem with hadoop-snappy
Posted by Colin McCabe <cm...@alumni.cmu.edu>.
I think what you're hitting is probably
https://issues.apache.org/jira/browse/HADOOP-8756. It was fixed in
branch-2 and branch-3, but not in the old releases.
If you want to work around the problem, try explicitly setting
LD_LIBRARY_PATH to include the directory that contains snappy.
cheers,
Colin
On Tue, Nov 6, 2012 at 1:30 PM, <al...@aim.com> wrote:
> When I used hadoop-1.0.4 without the installation of hadoop-snappy I saw in
> the logs
>
> WARN snappy.LoadSnappy: Snappy native library not loaded
>
> So, it was my understanding that snappy is not included with hadoop-1.0.4.
> After removing the libs I installed, I see again the same warning.
>
> Any ideas how to fix this issue?
>
> Thanks.
> Alex.
>
>
>
>
> -----Original Message-----
> From: Binglin Chang <de...@gmail.com>
> To: user <us...@hadoop.apache.org>
> Sent: Mon, Nov 5, 2012 6:07 pm
> Subject: Re: problem with hadoop-snappy
>
> I think hadoop-1.0.4 already have snappy included, you should not using
> other third party libraries.
>
>
>
> On Tue, Nov 6, 2012 at 9:10 AM, <al...@aim.com> wrote:
>>
>> Hello,
>>
>> I use hadoop-1.0.4 I have followed instruction to install hadoop-snappy
>> at
>> http://code.google.com/p/hadoop-snappy/
>>
>> When I run a mapred job I see
>>
>> FATAL org.apache.hadoop.mapred.TaskTracker: Task:
>> attempt_201211051656_0002_m_000000_3 - Killed :
>> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
>>
>> in task tracker log and the job fails.
>>
>> Any ideas how to fix this issue.
>>
>> Thanks in advance.
>> Alex.
>>
>
Re: problem with hadoop-snappy
Posted by Arpit Gupta <ar...@hortonworks.com>.
Alex
You need to make sure libsnappy.so is available in hadoop's java library path.
The way hadoop sets the java library path is that based on the jdk it will add either 32bit or 64bit libraries.
They need to be present in $HADOOP_HOME/lib/native/Linux-amd64-64 for 64bit jdk or $HADOOP_HOME/lib/native/Linux-i386-32 for 32 bit jdk. So create the libsnappy.so link in the appropriate directories on all your nodes restart your tasktrackers and submit your job again.
--
Arpit Gupta
Hortonworks Inc.
http://hortonworks.com/
On Nov 6, 2012, at 5:42 PM, alxsss@aim.com wrote:
> Yes, I have installed snappy-1.0.5.
>
> Here is the output of
> ls -l /usr/local/lib
> total 1468
> lrwxrwxrwx. 1 root root 52 Nov 5 16:00 libjvm.so -> /usr/java/jdk1.6.0_37/jre/lib/amd64/server/libjvm.so
> -rw-r--r--. 1 root root 946358 Nov 3 17:44 liblzo2.a
> -rwxr-xr-x. 1 root root 866 Nov 3 17:44 liblzo2.la
> -rw-r--r--. 1 root root 371174 Nov 5 15:18 libsnappy.a
> -rwxr-xr-x. 1 root root 957 Nov 5 15:18 libsnappy.la
> lrwxrwxrwx. 1 root root 18 Nov 5 15:18 libsnappy.so -> libsnappy.so.1.1.3
> lrwxrwxrwx. 1 root root 18 Nov 5 15:18 libsnappy.so.1 -> libsnappy.so.1.1.3
> -rwxr-xr-x. 1 root root 169393 Nov 5 15:18 libsnappy.so.1.1.3
>
>
>
> -----Original Message-----
> From: Binglin Chang <de...@gmail.com>
> To: user <us...@hadoop.apache.org>
> Sent: Tue, Nov 6, 2012 5:34 pm
> Subject: Re: problem with hadoop-snappy
>
> Have you installed snappy library (which should have libsnappy.so in /usr/lib or /usr/local/lib) first?
>
>
>
> On Wed, Nov 7, 2012 at 5:30 AM, <al...@aim.com> wrote:
> When I used hadoop-1.0.4 without the installation of hadoop-snappy I saw in the logs
>
> WARN snappy.LoadSnappy: Snappy native library not loaded
>
> So, it was my understanding that snappy is not included with hadoop-1.0.4. After removing the libs I installed, I see again the same warning.
>
> Any ideas how to fix this issue?
>
> Thanks.
> Alex.
>
>
>
>
> -----Original Message-----
> From: Binglin Chang <de...@gmail.com>
> To: user <us...@hadoop.apache.org>
> Sent: Mon, Nov 5, 2012 6:07 pm
> Subject: Re: problem with hadoop-snappy
>
> I think hadoop-1.0.4 already have snappy included, you should not using other third party libraries.
>
>
>
> On Tue, Nov 6, 2012 at 9:10 AM, <al...@aim.com> wrote:
> Hello,
>
> I use hadoop-1.0.4 I have followed instruction to install hadoop-snappy at
> http://code.google.com/p/hadoop-snappy/
>
> When I run a mapred job I see
>
> FATAL org.apache.hadoop.mapred.TaskTracker: Task: attempt_201211051656_0002_m_000000_3 - Killed : org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
>
> in task tracker log and the job fails.
>
> Any ideas how to fix this issue.
>
> Thanks in advance.
> Alex.
>
>
>
Re: problem with hadoop-snappy
Posted by Arpit Gupta <ar...@hortonworks.com>.
Alex
You need to make sure libsnappy.so is available in hadoop's java library path.
The way hadoop sets the java library path is that based on the jdk it will add either 32bit or 64bit libraries.
They need to be present in $HADOOP_HOME/lib/native/Linux-amd64-64 for 64bit jdk or $HADOOP_HOME/lib/native/Linux-i386-32 for 32 bit jdk. So create the libsnappy.so link in the appropriate directories on all your nodes restart your tasktrackers and submit your job again.
--
Arpit Gupta
Hortonworks Inc.
http://hortonworks.com/
On Nov 6, 2012, at 5:42 PM, alxsss@aim.com wrote:
> Yes, I have installed snappy-1.0.5.
>
> Here is the output of
> ls -l /usr/local/lib
> total 1468
> lrwxrwxrwx. 1 root root 52 Nov 5 16:00 libjvm.so -> /usr/java/jdk1.6.0_37/jre/lib/amd64/server/libjvm.so
> -rw-r--r--. 1 root root 946358 Nov 3 17:44 liblzo2.a
> -rwxr-xr-x. 1 root root 866 Nov 3 17:44 liblzo2.la
> -rw-r--r--. 1 root root 371174 Nov 5 15:18 libsnappy.a
> -rwxr-xr-x. 1 root root 957 Nov 5 15:18 libsnappy.la
> lrwxrwxrwx. 1 root root 18 Nov 5 15:18 libsnappy.so -> libsnappy.so.1.1.3
> lrwxrwxrwx. 1 root root 18 Nov 5 15:18 libsnappy.so.1 -> libsnappy.so.1.1.3
> -rwxr-xr-x. 1 root root 169393 Nov 5 15:18 libsnappy.so.1.1.3
>
>
>
> -----Original Message-----
> From: Binglin Chang <de...@gmail.com>
> To: user <us...@hadoop.apache.org>
> Sent: Tue, Nov 6, 2012 5:34 pm
> Subject: Re: problem with hadoop-snappy
>
> Have you installed snappy library (which should have libsnappy.so in /usr/lib or /usr/local/lib) first?
>
>
>
> On Wed, Nov 7, 2012 at 5:30 AM, <al...@aim.com> wrote:
> When I used hadoop-1.0.4 without the installation of hadoop-snappy I saw in the logs
>
> WARN snappy.LoadSnappy: Snappy native library not loaded
>
> So, it was my understanding that snappy is not included with hadoop-1.0.4. After removing the libs I installed, I see again the same warning.
>
> Any ideas how to fix this issue?
>
> Thanks.
> Alex.
>
>
>
>
> -----Original Message-----
> From: Binglin Chang <de...@gmail.com>
> To: user <us...@hadoop.apache.org>
> Sent: Mon, Nov 5, 2012 6:07 pm
> Subject: Re: problem with hadoop-snappy
>
> I think hadoop-1.0.4 already have snappy included, you should not using other third party libraries.
>
>
>
> On Tue, Nov 6, 2012 at 9:10 AM, <al...@aim.com> wrote:
> Hello,
>
> I use hadoop-1.0.4 I have followed instruction to install hadoop-snappy at
> http://code.google.com/p/hadoop-snappy/
>
> When I run a mapred job I see
>
> FATAL org.apache.hadoop.mapred.TaskTracker: Task: attempt_201211051656_0002_m_000000_3 - Killed : org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
>
> in task tracker log and the job fails.
>
> Any ideas how to fix this issue.
>
> Thanks in advance.
> Alex.
>
>
>
Re: problem with hadoop-snappy
Posted by Arpit Gupta <ar...@hortonworks.com>.
Alex
You need to make sure libsnappy.so is available in hadoop's java library path.
The way hadoop sets the java library path is that based on the jdk it will add either 32bit or 64bit libraries.
They need to be present in $HADOOP_HOME/lib/native/Linux-amd64-64 for 64bit jdk or $HADOOP_HOME/lib/native/Linux-i386-32 for 32 bit jdk. So create the libsnappy.so link in the appropriate directories on all your nodes restart your tasktrackers and submit your job again.
--
Arpit Gupta
Hortonworks Inc.
http://hortonworks.com/
On Nov 6, 2012, at 5:42 PM, alxsss@aim.com wrote:
> Yes, I have installed snappy-1.0.5.
>
> Here is the output of
> ls -l /usr/local/lib
> total 1468
> lrwxrwxrwx. 1 root root 52 Nov 5 16:00 libjvm.so -> /usr/java/jdk1.6.0_37/jre/lib/amd64/server/libjvm.so
> -rw-r--r--. 1 root root 946358 Nov 3 17:44 liblzo2.a
> -rwxr-xr-x. 1 root root 866 Nov 3 17:44 liblzo2.la
> -rw-r--r--. 1 root root 371174 Nov 5 15:18 libsnappy.a
> -rwxr-xr-x. 1 root root 957 Nov 5 15:18 libsnappy.la
> lrwxrwxrwx. 1 root root 18 Nov 5 15:18 libsnappy.so -> libsnappy.so.1.1.3
> lrwxrwxrwx. 1 root root 18 Nov 5 15:18 libsnappy.so.1 -> libsnappy.so.1.1.3
> -rwxr-xr-x. 1 root root 169393 Nov 5 15:18 libsnappy.so.1.1.3
>
>
>
> -----Original Message-----
> From: Binglin Chang <de...@gmail.com>
> To: user <us...@hadoop.apache.org>
> Sent: Tue, Nov 6, 2012 5:34 pm
> Subject: Re: problem with hadoop-snappy
>
> Have you installed snappy library (which should have libsnappy.so in /usr/lib or /usr/local/lib) first?
>
>
>
> On Wed, Nov 7, 2012 at 5:30 AM, <al...@aim.com> wrote:
> When I used hadoop-1.0.4 without the installation of hadoop-snappy I saw in the logs
>
> WARN snappy.LoadSnappy: Snappy native library not loaded
>
> So, it was my understanding that snappy is not included with hadoop-1.0.4. After removing the libs I installed, I see again the same warning.
>
> Any ideas how to fix this issue?
>
> Thanks.
> Alex.
>
>
>
>
> -----Original Message-----
> From: Binglin Chang <de...@gmail.com>
> To: user <us...@hadoop.apache.org>
> Sent: Mon, Nov 5, 2012 6:07 pm
> Subject: Re: problem with hadoop-snappy
>
> I think hadoop-1.0.4 already have snappy included, you should not using other third party libraries.
>
>
>
> On Tue, Nov 6, 2012 at 9:10 AM, <al...@aim.com> wrote:
> Hello,
>
> I use hadoop-1.0.4 I have followed instruction to install hadoop-snappy at
> http://code.google.com/p/hadoop-snappy/
>
> When I run a mapred job I see
>
> FATAL org.apache.hadoop.mapred.TaskTracker: Task: attempt_201211051656_0002_m_000000_3 - Killed : org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
>
> in task tracker log and the job fails.
>
> Any ideas how to fix this issue.
>
> Thanks in advance.
> Alex.
>
>
>
Re: problem with hadoop-snappy
Posted by Arpit Gupta <ar...@hortonworks.com>.
Alex
You need to make sure libsnappy.so is available in hadoop's java library path.
The way hadoop sets the java library path is that based on the jdk it will add either 32bit or 64bit libraries.
They need to be present in $HADOOP_HOME/lib/native/Linux-amd64-64 for 64bit jdk or $HADOOP_HOME/lib/native/Linux-i386-32 for 32 bit jdk. So create the libsnappy.so link in the appropriate directories on all your nodes restart your tasktrackers and submit your job again.
--
Arpit Gupta
Hortonworks Inc.
http://hortonworks.com/
On Nov 6, 2012, at 5:42 PM, alxsss@aim.com wrote:
> Yes, I have installed snappy-1.0.5.
>
> Here is the output of
> ls -l /usr/local/lib
> total 1468
> lrwxrwxrwx. 1 root root 52 Nov 5 16:00 libjvm.so -> /usr/java/jdk1.6.0_37/jre/lib/amd64/server/libjvm.so
> -rw-r--r--. 1 root root 946358 Nov 3 17:44 liblzo2.a
> -rwxr-xr-x. 1 root root 866 Nov 3 17:44 liblzo2.la
> -rw-r--r--. 1 root root 371174 Nov 5 15:18 libsnappy.a
> -rwxr-xr-x. 1 root root 957 Nov 5 15:18 libsnappy.la
> lrwxrwxrwx. 1 root root 18 Nov 5 15:18 libsnappy.so -> libsnappy.so.1.1.3
> lrwxrwxrwx. 1 root root 18 Nov 5 15:18 libsnappy.so.1 -> libsnappy.so.1.1.3
> -rwxr-xr-x. 1 root root 169393 Nov 5 15:18 libsnappy.so.1.1.3
>
>
>
> -----Original Message-----
> From: Binglin Chang <de...@gmail.com>
> To: user <us...@hadoop.apache.org>
> Sent: Tue, Nov 6, 2012 5:34 pm
> Subject: Re: problem with hadoop-snappy
>
> Have you installed snappy library (which should have libsnappy.so in /usr/lib or /usr/local/lib) first?
>
>
>
> On Wed, Nov 7, 2012 at 5:30 AM, <al...@aim.com> wrote:
> When I used hadoop-1.0.4 without the installation of hadoop-snappy I saw in the logs
>
> WARN snappy.LoadSnappy: Snappy native library not loaded
>
> So, it was my understanding that snappy is not included with hadoop-1.0.4. After removing the libs I installed, I see again the same warning.
>
> Any ideas how to fix this issue?
>
> Thanks.
> Alex.
>
>
>
>
> -----Original Message-----
> From: Binglin Chang <de...@gmail.com>
> To: user <us...@hadoop.apache.org>
> Sent: Mon, Nov 5, 2012 6:07 pm
> Subject: Re: problem with hadoop-snappy
>
> I think hadoop-1.0.4 already have snappy included, you should not using other third party libraries.
>
>
>
> On Tue, Nov 6, 2012 at 9:10 AM, <al...@aim.com> wrote:
> Hello,
>
> I use hadoop-1.0.4 I have followed instruction to install hadoop-snappy at
> http://code.google.com/p/hadoop-snappy/
>
> When I run a mapred job I see
>
> FATAL org.apache.hadoop.mapred.TaskTracker: Task: attempt_201211051656_0002_m_000000_3 - Killed : org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
>
> in task tracker log and the job fails.
>
> Any ideas how to fix this issue.
>
> Thanks in advance.
> Alex.
>
>
>
Re: problem with hadoop-snappy
Posted by al...@aim.com.
Yes, I have installed snappy-1.0.5.
Here is the output of
ls -l /usr/local/lib
total 1468
lrwxrwxrwx. 1 root root 52 Nov 5 16:00 libjvm.so -> /usr/java/jdk1.6.0_37/jre/lib/amd64/server/libjvm.so
-rw-r--r--. 1 root root 946358 Nov 3 17:44 liblzo2.a
-rwxr-xr-x. 1 root root 866 Nov 3 17:44 liblzo2.la
-rw-r--r--. 1 root root 371174 Nov 5 15:18 libsnappy.a
-rwxr-xr-x. 1 root root 957 Nov 5 15:18 libsnappy.la
lrwxrwxrwx. 1 root root 18 Nov 5 15:18 libsnappy.so -> libsnappy.so.1.1.3
lrwxrwxrwx. 1 root root 18 Nov 5 15:18 libsnappy.so.1 -> libsnappy.so.1.1.3
-rwxr-xr-x. 1 root root 169393 Nov 5 15:18 libsnappy.so.1.1.3
-----Original Message-----
From: Binglin Chang <de...@gmail.com>
To: user <us...@hadoop.apache.org>
Sent: Tue, Nov 6, 2012 5:34 pm
Subject: Re: problem with hadoop-snappy
Have you installed snappy library (which should have libsnappy.so in /usr/lib or /usr/local/lib) first?
On Wed, Nov 7, 2012 at 5:30 AM, <al...@aim.com> wrote:
When I used hadoop-1.0.4 without the installation of hadoop-snappy I saw in the logs
WARN snappy.LoadSnappy: Snappy native library not loaded
So, it was my understanding that snappy is not included with hadoop-1.0.4. After removing the libs I installed, I see again the same warning.
Any ideas how to fix this issue?
Thanks.
Alex.
-----Original Message-----
From: Binglin Chang <de...@gmail.com>
To: user <us...@hadoop.apache.org>
Sent: Mon, Nov 5, 2012 6:07 pm
Subject: Re: problem with hadoop-snappy
I think hadoop-1.0.4 already have snappy included, you should not using other third party libraries.
On Tue, Nov 6, 2012 at 9:10 AM, <al...@aim.com> wrote:
Hello,
I use hadoop-1.0.4 I have followed instruction to install hadoop-snappy at
http://code.google.com/p/hadoop-snappy/
When I run a mapred job I see
FATAL org.apache.hadoop.mapred.TaskTracker: Task: attempt_201211051656_0002_m_000000_3 - Killed : org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
in task tracker log and the job fails.
Any ideas how to fix this issue.
Thanks in advance.
Alex.
Re: problem with hadoop-snappy
Posted by al...@aim.com.
Yes, I have installed snappy-1.0.5.
Here is the output of
ls -l /usr/local/lib
total 1468
lrwxrwxrwx. 1 root root 52 Nov 5 16:00 libjvm.so -> /usr/java/jdk1.6.0_37/jre/lib/amd64/server/libjvm.so
-rw-r--r--. 1 root root 946358 Nov 3 17:44 liblzo2.a
-rwxr-xr-x. 1 root root 866 Nov 3 17:44 liblzo2.la
-rw-r--r--. 1 root root 371174 Nov 5 15:18 libsnappy.a
-rwxr-xr-x. 1 root root 957 Nov 5 15:18 libsnappy.la
lrwxrwxrwx. 1 root root 18 Nov 5 15:18 libsnappy.so -> libsnappy.so.1.1.3
lrwxrwxrwx. 1 root root 18 Nov 5 15:18 libsnappy.so.1 -> libsnappy.so.1.1.3
-rwxr-xr-x. 1 root root 169393 Nov 5 15:18 libsnappy.so.1.1.3
-----Original Message-----
From: Binglin Chang <de...@gmail.com>
To: user <us...@hadoop.apache.org>
Sent: Tue, Nov 6, 2012 5:34 pm
Subject: Re: problem with hadoop-snappy
Have you installed snappy library (which should have libsnappy.so in /usr/lib or /usr/local/lib) first?
On Wed, Nov 7, 2012 at 5:30 AM, <al...@aim.com> wrote:
When I used hadoop-1.0.4 without the installation of hadoop-snappy I saw in the logs
WARN snappy.LoadSnappy: Snappy native library not loaded
So, it was my understanding that snappy is not included with hadoop-1.0.4. After removing the libs I installed, I see again the same warning.
Any ideas how to fix this issue?
Thanks.
Alex.
-----Original Message-----
From: Binglin Chang <de...@gmail.com>
To: user <us...@hadoop.apache.org>
Sent: Mon, Nov 5, 2012 6:07 pm
Subject: Re: problem with hadoop-snappy
I think hadoop-1.0.4 already have snappy included, you should not using other third party libraries.
On Tue, Nov 6, 2012 at 9:10 AM, <al...@aim.com> wrote:
Hello,
I use hadoop-1.0.4 I have followed instruction to install hadoop-snappy at
http://code.google.com/p/hadoop-snappy/
When I run a mapred job I see
FATAL org.apache.hadoop.mapred.TaskTracker: Task: attempt_201211051656_0002_m_000000_3 - Killed : org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
in task tracker log and the job fails.
Any ideas how to fix this issue.
Thanks in advance.
Alex.
Re: problem with hadoop-snappy
Posted by al...@aim.com.
Yes, I have installed snappy-1.0.5.
Here is the output of
ls -l /usr/local/lib
total 1468
lrwxrwxrwx. 1 root root 52 Nov 5 16:00 libjvm.so -> /usr/java/jdk1.6.0_37/jre/lib/amd64/server/libjvm.so
-rw-r--r--. 1 root root 946358 Nov 3 17:44 liblzo2.a
-rwxr-xr-x. 1 root root 866 Nov 3 17:44 liblzo2.la
-rw-r--r--. 1 root root 371174 Nov 5 15:18 libsnappy.a
-rwxr-xr-x. 1 root root 957 Nov 5 15:18 libsnappy.la
lrwxrwxrwx. 1 root root 18 Nov 5 15:18 libsnappy.so -> libsnappy.so.1.1.3
lrwxrwxrwx. 1 root root 18 Nov 5 15:18 libsnappy.so.1 -> libsnappy.so.1.1.3
-rwxr-xr-x. 1 root root 169393 Nov 5 15:18 libsnappy.so.1.1.3
-----Original Message-----
From: Binglin Chang <de...@gmail.com>
To: user <us...@hadoop.apache.org>
Sent: Tue, Nov 6, 2012 5:34 pm
Subject: Re: problem with hadoop-snappy
Have you installed snappy library (which should have libsnappy.so in /usr/lib or /usr/local/lib) first?
On Wed, Nov 7, 2012 at 5:30 AM, <al...@aim.com> wrote:
When I used hadoop-1.0.4 without the installation of hadoop-snappy I saw in the logs
WARN snappy.LoadSnappy: Snappy native library not loaded
So, it was my understanding that snappy is not included with hadoop-1.0.4. After removing the libs I installed, I see again the same warning.
Any ideas how to fix this issue?
Thanks.
Alex.
-----Original Message-----
From: Binglin Chang <de...@gmail.com>
To: user <us...@hadoop.apache.org>
Sent: Mon, Nov 5, 2012 6:07 pm
Subject: Re: problem with hadoop-snappy
I think hadoop-1.0.4 already have snappy included, you should not using other third party libraries.
On Tue, Nov 6, 2012 at 9:10 AM, <al...@aim.com> wrote:
Hello,
I use hadoop-1.0.4 I have followed instruction to install hadoop-snappy at
http://code.google.com/p/hadoop-snappy/
When I run a mapred job I see
FATAL org.apache.hadoop.mapred.TaskTracker: Task: attempt_201211051656_0002_m_000000_3 - Killed : org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
in task tracker log and the job fails.
Any ideas how to fix this issue.
Thanks in advance.
Alex.
Re: problem with hadoop-snappy
Posted by al...@aim.com.
Yes, I have installed snappy-1.0.5.
Here is the output of
ls -l /usr/local/lib
total 1468
lrwxrwxrwx. 1 root root 52 Nov 5 16:00 libjvm.so -> /usr/java/jdk1.6.0_37/jre/lib/amd64/server/libjvm.so
-rw-r--r--. 1 root root 946358 Nov 3 17:44 liblzo2.a
-rwxr-xr-x. 1 root root 866 Nov 3 17:44 liblzo2.la
-rw-r--r--. 1 root root 371174 Nov 5 15:18 libsnappy.a
-rwxr-xr-x. 1 root root 957 Nov 5 15:18 libsnappy.la
lrwxrwxrwx. 1 root root 18 Nov 5 15:18 libsnappy.so -> libsnappy.so.1.1.3
lrwxrwxrwx. 1 root root 18 Nov 5 15:18 libsnappy.so.1 -> libsnappy.so.1.1.3
-rwxr-xr-x. 1 root root 169393 Nov 5 15:18 libsnappy.so.1.1.3
-----Original Message-----
From: Binglin Chang <de...@gmail.com>
To: user <us...@hadoop.apache.org>
Sent: Tue, Nov 6, 2012 5:34 pm
Subject: Re: problem with hadoop-snappy
Have you installed snappy library (which should have libsnappy.so in /usr/lib or /usr/local/lib) first?
On Wed, Nov 7, 2012 at 5:30 AM, <al...@aim.com> wrote:
When I used hadoop-1.0.4 without the installation of hadoop-snappy I saw in the logs
WARN snappy.LoadSnappy: Snappy native library not loaded
So, it was my understanding that snappy is not included with hadoop-1.0.4. After removing the libs I installed, I see again the same warning.
Any ideas how to fix this issue?
Thanks.
Alex.
-----Original Message-----
From: Binglin Chang <de...@gmail.com>
To: user <us...@hadoop.apache.org>
Sent: Mon, Nov 5, 2012 6:07 pm
Subject: Re: problem with hadoop-snappy
I think hadoop-1.0.4 already have snappy included, you should not using other third party libraries.
On Tue, Nov 6, 2012 at 9:10 AM, <al...@aim.com> wrote:
Hello,
I use hadoop-1.0.4 I have followed instruction to install hadoop-snappy at
http://code.google.com/p/hadoop-snappy/
When I run a mapred job I see
FATAL org.apache.hadoop.mapred.TaskTracker: Task: attempt_201211051656_0002_m_000000_3 - Killed : org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
in task tracker log and the job fails.
Any ideas how to fix this issue.
Thanks in advance.
Alex.
Re: problem with hadoop-snappy
Posted by Binglin Chang <de...@gmail.com>.
Have you installed snappy library (which should have libsnappy.so in
/usr/lib or /usr/local/lib) first?
On Wed, Nov 7, 2012 at 5:30 AM, <al...@aim.com> wrote:
> When I used hadoop-1.0.4 without the installation of hadoop-snappy I sawin the logs
>
> WARN snappy.LoadSnappy: Snappy native library not loaded
>
> So, it was my understanding that snappy is not included with
> hadoop-1.0.4. After removing the libs I installed, I see again the same
> warning.
>
> Any ideas how to fix this issue?
>
> Thanks.
> Alex.
>
>
>
>
> -----Original Message-----
> From: Binglin Chang <de...@gmail.com>
> To: user <us...@hadoop.apache.org>
> Sent: Mon, Nov 5, 2012 6:07 pm
> Subject: Re: problem with hadoop-snappy
>
> I think hadoop-1.0.4 already have snappy included, you should not using
> other third party libraries.
>
>
>
> On Tue, Nov 6, 2012 at 9:10 AM, <al...@aim.com> wrote:
>
>> Hello,
>>
>> I use hadoop-1.0.4 I have followed instruction to install hadoop-snappy
>> at
>> http://code.google.com/p/hadoop-snappy/
>>
>> When I run a mapred job I see
>>
>> FATAL org.apache.hadoop.mapred.TaskTracker: Task:
>> attempt_201211051656_0002_m_000000_3 - Killed :
>> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
>>
>> in task tracker log and the job fails.
>>
>> Any ideas how to fix this issue.
>>
>> Thanks in advance.
>> Alex.
>>
>>
>
Re: problem with hadoop-snappy
Posted by Binglin Chang <de...@gmail.com>.
Have you installed snappy library (which should have libsnappy.so in
/usr/lib or /usr/local/lib) first?
On Wed, Nov 7, 2012 at 5:30 AM, <al...@aim.com> wrote:
> When I used hadoop-1.0.4 without the installation of hadoop-snappy I sawin the logs
>
> WARN snappy.LoadSnappy: Snappy native library not loaded
>
> So, it was my understanding that snappy is not included with
> hadoop-1.0.4. After removing the libs I installed, I see again the same
> warning.
>
> Any ideas how to fix this issue?
>
> Thanks.
> Alex.
>
>
>
>
> -----Original Message-----
> From: Binglin Chang <de...@gmail.com>
> To: user <us...@hadoop.apache.org>
> Sent: Mon, Nov 5, 2012 6:07 pm
> Subject: Re: problem with hadoop-snappy
>
> I think hadoop-1.0.4 already have snappy included, you should not using
> other third party libraries.
>
>
>
> On Tue, Nov 6, 2012 at 9:10 AM, <al...@aim.com> wrote:
>
>> Hello,
>>
>> I use hadoop-1.0.4 I have followed instruction to install hadoop-snappy
>> at
>> http://code.google.com/p/hadoop-snappy/
>>
>> When I run a mapred job I see
>>
>> FATAL org.apache.hadoop.mapred.TaskTracker: Task:
>> attempt_201211051656_0002_m_000000_3 - Killed :
>> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
>>
>> in task tracker log and the job fails.
>>
>> Any ideas how to fix this issue.
>>
>> Thanks in advance.
>> Alex.
>>
>>
>
Re: problem with hadoop-snappy
Posted by Colin McCabe <cm...@alumni.cmu.edu>.
I think what you're hitting is probably
https://issues.apache.org/jira/browse/HADOOP-8756. It was fixed in
branch-2 and branch-3, but not in the old releases.
If you want to work around the problem, try explicitly setting
LD_LIBRARY_PATH to include the directory that contains snappy.
cheers,
Colin
On Tue, Nov 6, 2012 at 1:30 PM, <al...@aim.com> wrote:
> When I used hadoop-1.0.4 without the installation of hadoop-snappy I saw in
> the logs
>
> WARN snappy.LoadSnappy: Snappy native library not loaded
>
> So, it was my understanding that snappy is not included with hadoop-1.0.4.
> After removing the libs I installed, I see again the same warning.
>
> Any ideas how to fix this issue?
>
> Thanks.
> Alex.
>
>
>
>
> -----Original Message-----
> From: Binglin Chang <de...@gmail.com>
> To: user <us...@hadoop.apache.org>
> Sent: Mon, Nov 5, 2012 6:07 pm
> Subject: Re: problem with hadoop-snappy
>
> I think hadoop-1.0.4 already have snappy included, you should not using
> other third party libraries.
>
>
>
> On Tue, Nov 6, 2012 at 9:10 AM, <al...@aim.com> wrote:
>>
>> Hello,
>>
>> I use hadoop-1.0.4 I have followed instruction to install hadoop-snappy
>> at
>> http://code.google.com/p/hadoop-snappy/
>>
>> When I run a mapred job I see
>>
>> FATAL org.apache.hadoop.mapred.TaskTracker: Task:
>> attempt_201211051656_0002_m_000000_3 - Killed :
>> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
>>
>> in task tracker log and the job fails.
>>
>> Any ideas how to fix this issue.
>>
>> Thanks in advance.
>> Alex.
>>
>
Re: problem with hadoop-snappy
Posted by Binglin Chang <de...@gmail.com>.
Have you installed snappy library (which should have libsnappy.so in
/usr/lib or /usr/local/lib) first?
On Wed, Nov 7, 2012 at 5:30 AM, <al...@aim.com> wrote:
> When I used hadoop-1.0.4 without the installation of hadoop-snappy I sawin the logs
>
> WARN snappy.LoadSnappy: Snappy native library not loaded
>
> So, it was my understanding that snappy is not included with
> hadoop-1.0.4. After removing the libs I installed, I see again the same
> warning.
>
> Any ideas how to fix this issue?
>
> Thanks.
> Alex.
>
>
>
>
> -----Original Message-----
> From: Binglin Chang <de...@gmail.com>
> To: user <us...@hadoop.apache.org>
> Sent: Mon, Nov 5, 2012 6:07 pm
> Subject: Re: problem with hadoop-snappy
>
> I think hadoop-1.0.4 already have snappy included, you should not using
> other third party libraries.
>
>
>
> On Tue, Nov 6, 2012 at 9:10 AM, <al...@aim.com> wrote:
>
>> Hello,
>>
>> I use hadoop-1.0.4 I have followed instruction to install hadoop-snappy
>> at
>> http://code.google.com/p/hadoop-snappy/
>>
>> When I run a mapred job I see
>>
>> FATAL org.apache.hadoop.mapred.TaskTracker: Task:
>> attempt_201211051656_0002_m_000000_3 - Killed :
>> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
>>
>> in task tracker log and the job fails.
>>
>> Any ideas how to fix this issue.
>>
>> Thanks in advance.
>> Alex.
>>
>>
>
Re: problem with hadoop-snappy
Posted by Binglin Chang <de...@gmail.com>.
Have you installed snappy library (which should have libsnappy.so in
/usr/lib or /usr/local/lib) first?
On Wed, Nov 7, 2012 at 5:30 AM, <al...@aim.com> wrote:
> When I used hadoop-1.0.4 without the installation of hadoop-snappy I sawin the logs
>
> WARN snappy.LoadSnappy: Snappy native library not loaded
>
> So, it was my understanding that snappy is not included with
> hadoop-1.0.4. After removing the libs I installed, I see again the same
> warning.
>
> Any ideas how to fix this issue?
>
> Thanks.
> Alex.
>
>
>
>
> -----Original Message-----
> From: Binglin Chang <de...@gmail.com>
> To: user <us...@hadoop.apache.org>
> Sent: Mon, Nov 5, 2012 6:07 pm
> Subject: Re: problem with hadoop-snappy
>
> I think hadoop-1.0.4 already have snappy included, you should not using
> other third party libraries.
>
>
>
> On Tue, Nov 6, 2012 at 9:10 AM, <al...@aim.com> wrote:
>
>> Hello,
>>
>> I use hadoop-1.0.4 I have followed instruction to install hadoop-snappy
>> at
>> http://code.google.com/p/hadoop-snappy/
>>
>> When I run a mapred job I see
>>
>> FATAL org.apache.hadoop.mapred.TaskTracker: Task:
>> attempt_201211051656_0002_m_000000_3 - Killed :
>> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
>>
>> in task tracker log and the job fails.
>>
>> Any ideas how to fix this issue.
>>
>> Thanks in advance.
>> Alex.
>>
>>
>
Re: problem with hadoop-snappy
Posted by Colin McCabe <cm...@alumni.cmu.edu>.
I think what you're hitting is probably
https://issues.apache.org/jira/browse/HADOOP-8756. It was fixed in
branch-2 and branch-3, but not in the old releases.
If you want to work around the problem, try explicitly setting
LD_LIBRARY_PATH to include the directory that contains snappy.
cheers,
Colin
On Tue, Nov 6, 2012 at 1:30 PM, <al...@aim.com> wrote:
> When I used hadoop-1.0.4 without the installation of hadoop-snappy I saw in
> the logs
>
> WARN snappy.LoadSnappy: Snappy native library not loaded
>
> So, it was my understanding that snappy is not included with hadoop-1.0.4.
> After removing the libs I installed, I see again the same warning.
>
> Any ideas how to fix this issue?
>
> Thanks.
> Alex.
>
>
>
>
> -----Original Message-----
> From: Binglin Chang <de...@gmail.com>
> To: user <us...@hadoop.apache.org>
> Sent: Mon, Nov 5, 2012 6:07 pm
> Subject: Re: problem with hadoop-snappy
>
> I think hadoop-1.0.4 already have snappy included, you should not using
> other third party libraries.
>
>
>
> On Tue, Nov 6, 2012 at 9:10 AM, <al...@aim.com> wrote:
>>
>> Hello,
>>
>> I use hadoop-1.0.4 I have followed instruction to install hadoop-snappy
>> at
>> http://code.google.com/p/hadoop-snappy/
>>
>> When I run a mapred job I see
>>
>> FATAL org.apache.hadoop.mapred.TaskTracker: Task:
>> attempt_201211051656_0002_m_000000_3 - Killed :
>> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
>>
>> in task tracker log and the job fails.
>>
>> Any ideas how to fix this issue.
>>
>> Thanks in advance.
>> Alex.
>>
>
Re: problem with hadoop-snappy
Posted by al...@aim.com.
When I used hadoop-1.0.4 without the installation of hadoop-snappy I saw in the logs
WARN snappy.LoadSnappy: Snappy native library not loaded
So, it was my understanding that snappy is not included with hadoop-1.0.4. After removing the libs I installed, I see again the same warning.
Any ideas how to fix this issue?
Thanks.
Alex.
-----Original Message-----
From: Binglin Chang <de...@gmail.com>
To: user <us...@hadoop.apache.org>
Sent: Mon, Nov 5, 2012 6:07 pm
Subject: Re: problem with hadoop-snappy
I think hadoop-1.0.4 already have snappy included, you should not using other third party libraries.
On Tue, Nov 6, 2012 at 9:10 AM, <al...@aim.com> wrote:
Hello,
I use hadoop-1.0.4 I have followed instruction to install hadoop-snappy at
http://code.google.com/p/hadoop-snappy/
When I run a mapred job I see
FATAL org.apache.hadoop.mapred.TaskTracker: Task: attempt_201211051656_0002_m_000000_3 - Killed : org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
in task tracker log and the job fails.
Any ideas how to fix this issue.
Thanks in advance.
Alex.
Re: problem with hadoop-snappy
Posted by al...@aim.com.
When I used hadoop-1.0.4 without the installation of hadoop-snappy I saw in the logs
WARN snappy.LoadSnappy: Snappy native library not loaded
So, it was my understanding that snappy is not included with hadoop-1.0.4. After removing the libs I installed, I see again the same warning.
Any ideas how to fix this issue?
Thanks.
Alex.
-----Original Message-----
From: Binglin Chang <de...@gmail.com>
To: user <us...@hadoop.apache.org>
Sent: Mon, Nov 5, 2012 6:07 pm
Subject: Re: problem with hadoop-snappy
I think hadoop-1.0.4 already have snappy included, you should not using other third party libraries.
On Tue, Nov 6, 2012 at 9:10 AM, <al...@aim.com> wrote:
Hello,
I use hadoop-1.0.4 I have followed instruction to install hadoop-snappy at
http://code.google.com/p/hadoop-snappy/
When I run a mapred job I see
FATAL org.apache.hadoop.mapred.TaskTracker: Task: attempt_201211051656_0002_m_000000_3 - Killed : org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
in task tracker log and the job fails.
Any ideas how to fix this issue.
Thanks in advance.
Alex.
Re: problem with hadoop-snappy
Posted by al...@aim.com.
When I used hadoop-1.0.4 without the installation of hadoop-snappy I saw in the logs
WARN snappy.LoadSnappy: Snappy native library not loaded
So, it was my understanding that snappy is not included with hadoop-1.0.4. After removing the libs I installed, I see again the same warning.
Any ideas how to fix this issue?
Thanks.
Alex.
-----Original Message-----
From: Binglin Chang <de...@gmail.com>
To: user <us...@hadoop.apache.org>
Sent: Mon, Nov 5, 2012 6:07 pm
Subject: Re: problem with hadoop-snappy
I think hadoop-1.0.4 already have snappy included, you should not using other third party libraries.
On Tue, Nov 6, 2012 at 9:10 AM, <al...@aim.com> wrote:
Hello,
I use hadoop-1.0.4 I have followed instruction to install hadoop-snappy at
http://code.google.com/p/hadoop-snappy/
When I run a mapred job I see
FATAL org.apache.hadoop.mapred.TaskTracker: Task: attempt_201211051656_0002_m_000000_3 - Killed : org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
in task tracker log and the job fails.
Any ideas how to fix this issue.
Thanks in advance.
Alex.
Re: problem with hadoop-snappy
Posted by al...@aim.com.
When I used hadoop-1.0.4 without the installation of hadoop-snappy I saw in the logs
WARN snappy.LoadSnappy: Snappy native library not loaded
So, it was my understanding that snappy is not included with hadoop-1.0.4. After removing the libs I installed, I see again the same warning.
Any ideas how to fix this issue?
Thanks.
Alex.
-----Original Message-----
From: Binglin Chang <de...@gmail.com>
To: user <us...@hadoop.apache.org>
Sent: Mon, Nov 5, 2012 6:07 pm
Subject: Re: problem with hadoop-snappy
I think hadoop-1.0.4 already have snappy included, you should not using other third party libraries.
On Tue, Nov 6, 2012 at 9:10 AM, <al...@aim.com> wrote:
Hello,
I use hadoop-1.0.4 I have followed instruction to install hadoop-snappy at
http://code.google.com/p/hadoop-snappy/
When I run a mapred job I see
FATAL org.apache.hadoop.mapred.TaskTracker: Task: attempt_201211051656_0002_m_000000_3 - Killed : org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
in task tracker log and the job fails.
Any ideas how to fix this issue.
Thanks in advance.
Alex.
Re: problem with hadoop-snappy
Posted by Binglin Chang <de...@gmail.com>.
I think hadoop-1.0.4 already have snappy included, you should not using
other third party libraries.
On Tue, Nov 6, 2012 at 9:10 AM, <al...@aim.com> wrote:
> Hello,
>
> I use hadoop-1.0.4 I have followed instruction to install hadoop-snappy
> at
> http://code.google.com/p/hadoop-snappy/
>
> When I run a mapred job I see
>
> FATAL org.apache.hadoop.mapred.TaskTracker: Task:
> attempt_201211051656_0002_m_000000_3 - Killed :
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
>
> in task tracker log and the job fails.
>
> Any ideas how to fix this issue.
>
> Thanks in advance.
> Alex.
>
>
Re: problem with hadoop-snappy
Posted by Binglin Chang <de...@gmail.com>.
I think hadoop-1.0.4 already have snappy included, you should not using
other third party libraries.
On Tue, Nov 6, 2012 at 9:10 AM, <al...@aim.com> wrote:
> Hello,
>
> I use hadoop-1.0.4 I have followed instruction to install hadoop-snappy
> at
> http://code.google.com/p/hadoop-snappy/
>
> When I run a mapred job I see
>
> FATAL org.apache.hadoop.mapred.TaskTracker: Task:
> attempt_201211051656_0002_m_000000_3 - Killed :
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
>
> in task tracker log and the job fails.
>
> Any ideas how to fix this issue.
>
> Thanks in advance.
> Alex.
>
>
Re: problem with hadoop-snappy
Posted by Binglin Chang <de...@gmail.com>.
I think hadoop-1.0.4 already have snappy included, you should not using
other third party libraries.
On Tue, Nov 6, 2012 at 9:10 AM, <al...@aim.com> wrote:
> Hello,
>
> I use hadoop-1.0.4 I have followed instruction to install hadoop-snappy
> at
> http://code.google.com/p/hadoop-snappy/
>
> When I run a mapred job I see
>
> FATAL org.apache.hadoop.mapred.TaskTracker: Task:
> attempt_201211051656_0002_m_000000_3 - Killed :
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
>
> in task tracker log and the job fails.
>
> Any ideas how to fix this issue.
>
> Thanks in advance.
> Alex.
>
>
Re: problem with hadoop-snappy
Posted by Binglin Chang <de...@gmail.com>.
I think hadoop-1.0.4 already have snappy included, you should not using
other third party libraries.
On Tue, Nov 6, 2012 at 9:10 AM, <al...@aim.com> wrote:
> Hello,
>
> I use hadoop-1.0.4 I have followed instruction to install hadoop-snappy
> at
> http://code.google.com/p/hadoop-snappy/
>
> When I run a mapred job I see
>
> FATAL org.apache.hadoop.mapred.TaskTracker: Task:
> attempt_201211051656_0002_m_000000_3 - Killed :
> org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
>
> in task tracker log and the job fails.
>
> Any ideas how to fix this issue.
>
> Thanks in advance.
> Alex.
>
>