You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Yves Weissig <we...@uni-mainz.de> on 2014/02/11 13:16:04 UTC
Hadoop native and snappy library
Hi list,
I'm trying to enable the Hadoop native library and the snappy library
for compression in Hadoop 2.2.0, but I always end up with:
./hadoop/bin/hadoop checknative -a
Native library checking:
hadoop: false
zlib: false
snappy: false
lz4: false
bzip2: false
I compiled hadoop-2.2.0-src from scratch for x64 and put the resulting
.so in hadoop/lib/native/. I also compiled snappy from scratch and put
it there. In a different approach I installed snappy via sudo apt-get
and then linked the resulting .so to hadoop/lib/native/libsnappy.so,
still no luck.
What is going on here? Why won't Hadoop find my native libraries? Is
there any log where I can check what went wrong during loading?
Best regards
Yves
RE: Hadoop native and snappy library
Posted by java8964 <ja...@hotmail.com>.
Where do you compile your libhadoop.so.1.0.0?
It is more like that you compiled libhadoop.so.1.0.0 in a environment with glibc 2.14, but tried to use it in an environment only have glibc 2.12.
If you are using a hadoop compiled by yourself, then it is best to compile in an environment matching with your production.
Yong
> Date: Tue, 11 Feb 2014 14:12:34 +0100
> From: weissig@uni-mainz.de
> To: user@hadoop.apache.org
> Subject: Re: Hadoop native and snappy library
>
> Thanks... after looking at the sourcecode of the native library loading
> process and activating the DEBUG messages I found this in the "hadoop.log":
>
> 2014-02-11 14:01:14,084 DEBUG org.apache.hadoop.util.NativeCodeLoader:
> Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError:
> /hadoop-2.2.0/lib/native/libhadoop.so.1.0.0: /lib64/libc.so.6: version
> `GLIBC_2.14' not found (required by
> /hadoop-2.2.0/lib/native/libhadoop.so.1.0.0)
>
> Is it possible to compile the native library with glibc 2.12, because at
> the moment I am not able to upgrade glibc on the servers which run Hadoop.
>
> Best regards
> Yves
>
> On 11.02.2014 13:29, Ted Yu wrote:
> > See https://issues.apache.org/jira/browse/HBASE-10303
> >
> > And https://hbase.apache.org/book.html#snappy.compression
> >
> > Cheers
> >
> > On Feb 11, 2014, at 4:16 AM, Yves Weissig <weissig@uni-mainz.de
> > <ma...@uni-mainz.de>> wrote:
> >
> >> Hi list,
> >>
> >> I'm trying to enable the Hadoop native library and the snappy library
> >> for compression in Hadoop 2.2.0, but I always end up with:
> >>
> >> ./hadoop/bin/hadoop checknative -a
> >> Native library checking:
> >> hadoop: false
> >> zlib: false
> >> snappy: false
> >> lz4: false
> >> bzip2: false
> >>
> >> I compiled hadoop-2.2.0-src from scratch for x64 and put the resulting
> >> .so in hadoop/lib/native/. I also compiled snappy from scratch and put
> >> it there. In a different approach I installed snappy via sudo apt-get
> >> and then linked the resulting .so to hadoop/lib/native/libsnappy.so,
> >> still no luck.
> >>
> >> What is going on here? Why won't Hadoop find my native libraries? Is
> >> there any log where I can check what went wrong during loading?
> >>
> >> Best regards
> >> Yves
> >>
>
RE: Hadoop native and snappy library
Posted by java8964 <ja...@hotmail.com>.
Where do you compile your libhadoop.so.1.0.0?
It is more like that you compiled libhadoop.so.1.0.0 in a environment with glibc 2.14, but tried to use it in an environment only have glibc 2.12.
If you are using a hadoop compiled by yourself, then it is best to compile in an environment matching with your production.
Yong
> Date: Tue, 11 Feb 2014 14:12:34 +0100
> From: weissig@uni-mainz.de
> To: user@hadoop.apache.org
> Subject: Re: Hadoop native and snappy library
>
> Thanks... after looking at the sourcecode of the native library loading
> process and activating the DEBUG messages I found this in the "hadoop.log":
>
> 2014-02-11 14:01:14,084 DEBUG org.apache.hadoop.util.NativeCodeLoader:
> Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError:
> /hadoop-2.2.0/lib/native/libhadoop.so.1.0.0: /lib64/libc.so.6: version
> `GLIBC_2.14' not found (required by
> /hadoop-2.2.0/lib/native/libhadoop.so.1.0.0)
>
> Is it possible to compile the native library with glibc 2.12, because at
> the moment I am not able to upgrade glibc on the servers which run Hadoop.
>
> Best regards
> Yves
>
> On 11.02.2014 13:29, Ted Yu wrote:
> > See https://issues.apache.org/jira/browse/HBASE-10303
> >
> > And https://hbase.apache.org/book.html#snappy.compression
> >
> > Cheers
> >
> > On Feb 11, 2014, at 4:16 AM, Yves Weissig <weissig@uni-mainz.de
> > <ma...@uni-mainz.de>> wrote:
> >
> >> Hi list,
> >>
> >> I'm trying to enable the Hadoop native library and the snappy library
> >> for compression in Hadoop 2.2.0, but I always end up with:
> >>
> >> ./hadoop/bin/hadoop checknative -a
> >> Native library checking:
> >> hadoop: false
> >> zlib: false
> >> snappy: false
> >> lz4: false
> >> bzip2: false
> >>
> >> I compiled hadoop-2.2.0-src from scratch for x64 and put the resulting
> >> .so in hadoop/lib/native/. I also compiled snappy from scratch and put
> >> it there. In a different approach I installed snappy via sudo apt-get
> >> and then linked the resulting .so to hadoop/lib/native/libsnappy.so,
> >> still no luck.
> >>
> >> What is going on here? Why won't Hadoop find my native libraries? Is
> >> there any log where I can check what went wrong during loading?
> >>
> >> Best regards
> >> Yves
> >>
>
RE: Hadoop native and snappy library
Posted by java8964 <ja...@hotmail.com>.
Where do you compile your libhadoop.so.1.0.0?
It is more like that you compiled libhadoop.so.1.0.0 in a environment with glibc 2.14, but tried to use it in an environment only have glibc 2.12.
If you are using a hadoop compiled by yourself, then it is best to compile in an environment matching with your production.
Yong
> Date: Tue, 11 Feb 2014 14:12:34 +0100
> From: weissig@uni-mainz.de
> To: user@hadoop.apache.org
> Subject: Re: Hadoop native and snappy library
>
> Thanks... after looking at the sourcecode of the native library loading
> process and activating the DEBUG messages I found this in the "hadoop.log":
>
> 2014-02-11 14:01:14,084 DEBUG org.apache.hadoop.util.NativeCodeLoader:
> Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError:
> /hadoop-2.2.0/lib/native/libhadoop.so.1.0.0: /lib64/libc.so.6: version
> `GLIBC_2.14' not found (required by
> /hadoop-2.2.0/lib/native/libhadoop.so.1.0.0)
>
> Is it possible to compile the native library with glibc 2.12, because at
> the moment I am not able to upgrade glibc on the servers which run Hadoop.
>
> Best regards
> Yves
>
> On 11.02.2014 13:29, Ted Yu wrote:
> > See https://issues.apache.org/jira/browse/HBASE-10303
> >
> > And https://hbase.apache.org/book.html#snappy.compression
> >
> > Cheers
> >
> > On Feb 11, 2014, at 4:16 AM, Yves Weissig <weissig@uni-mainz.de
> > <ma...@uni-mainz.de>> wrote:
> >
> >> Hi list,
> >>
> >> I'm trying to enable the Hadoop native library and the snappy library
> >> for compression in Hadoop 2.2.0, but I always end up with:
> >>
> >> ./hadoop/bin/hadoop checknative -a
> >> Native library checking:
> >> hadoop: false
> >> zlib: false
> >> snappy: false
> >> lz4: false
> >> bzip2: false
> >>
> >> I compiled hadoop-2.2.0-src from scratch for x64 and put the resulting
> >> .so in hadoop/lib/native/. I also compiled snappy from scratch and put
> >> it there. In a different approach I installed snappy via sudo apt-get
> >> and then linked the resulting .so to hadoop/lib/native/libsnappy.so,
> >> still no luck.
> >>
> >> What is going on here? Why won't Hadoop find my native libraries? Is
> >> there any log where I can check what went wrong during loading?
> >>
> >> Best regards
> >> Yves
> >>
>
RE: Hadoop native and snappy library
Posted by java8964 <ja...@hotmail.com>.
Where do you compile your libhadoop.so.1.0.0?
It is more like that you compiled libhadoop.so.1.0.0 in a environment with glibc 2.14, but tried to use it in an environment only have glibc 2.12.
If you are using a hadoop compiled by yourself, then it is best to compile in an environment matching with your production.
Yong
> Date: Tue, 11 Feb 2014 14:12:34 +0100
> From: weissig@uni-mainz.de
> To: user@hadoop.apache.org
> Subject: Re: Hadoop native and snappy library
>
> Thanks... after looking at the sourcecode of the native library loading
> process and activating the DEBUG messages I found this in the "hadoop.log":
>
> 2014-02-11 14:01:14,084 DEBUG org.apache.hadoop.util.NativeCodeLoader:
> Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError:
> /hadoop-2.2.0/lib/native/libhadoop.so.1.0.0: /lib64/libc.so.6: version
> `GLIBC_2.14' not found (required by
> /hadoop-2.2.0/lib/native/libhadoop.so.1.0.0)
>
> Is it possible to compile the native library with glibc 2.12, because at
> the moment I am not able to upgrade glibc on the servers which run Hadoop.
>
> Best regards
> Yves
>
> On 11.02.2014 13:29, Ted Yu wrote:
> > See https://issues.apache.org/jira/browse/HBASE-10303
> >
> > And https://hbase.apache.org/book.html#snappy.compression
> >
> > Cheers
> >
> > On Feb 11, 2014, at 4:16 AM, Yves Weissig <weissig@uni-mainz.de
> > <ma...@uni-mainz.de>> wrote:
> >
> >> Hi list,
> >>
> >> I'm trying to enable the Hadoop native library and the snappy library
> >> for compression in Hadoop 2.2.0, but I always end up with:
> >>
> >> ./hadoop/bin/hadoop checknative -a
> >> Native library checking:
> >> hadoop: false
> >> zlib: false
> >> snappy: false
> >> lz4: false
> >> bzip2: false
> >>
> >> I compiled hadoop-2.2.0-src from scratch for x64 and put the resulting
> >> .so in hadoop/lib/native/. I also compiled snappy from scratch and put
> >> it there. In a different approach I installed snappy via sudo apt-get
> >> and then linked the resulting .so to hadoop/lib/native/libsnappy.so,
> >> still no luck.
> >>
> >> What is going on here? Why won't Hadoop find my native libraries? Is
> >> there any log where I can check what went wrong during loading?
> >>
> >> Best regards
> >> Yves
> >>
>
Re: Hadoop native and snappy library
Posted by Yves Weissig <we...@uni-mainz.de>.
Thanks... after looking at the sourcecode of the native library loading
process and activating the DEBUG messages I found this in the "hadoop.log":
2014-02-11 14:01:14,084 DEBUG org.apache.hadoop.util.NativeCodeLoader:
Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError:
/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0: /lib64/libc.so.6: version
`GLIBC_2.14' not found (required by
/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0)
Is it possible to compile the native library with glibc 2.12, because at
the moment I am not able to upgrade glibc on the servers which run Hadoop.
Best regards
Yves
On 11.02.2014 13:29, Ted Yu wrote:
> See https://issues.apache.org/jira/browse/HBASE-10303
>
> And https://hbase.apache.org/book.html#snappy.compression
>
> Cheers
>
> On Feb 11, 2014, at 4:16 AM, Yves Weissig <weissig@uni-mainz.de
> <ma...@uni-mainz.de>> wrote:
>
>> Hi list,
>>
>> I'm trying to enable the Hadoop native library and the snappy library
>> for compression in Hadoop 2.2.0, but I always end up with:
>>
>> ./hadoop/bin/hadoop checknative -a
>> Native library checking:
>> hadoop: false
>> zlib: false
>> snappy: false
>> lz4: false
>> bzip2: false
>>
>> I compiled hadoop-2.2.0-src from scratch for x64 and put the resulting
>> .so in hadoop/lib/native/. I also compiled snappy from scratch and put
>> it there. In a different approach I installed snappy via sudo apt-get
>> and then linked the resulting .so to hadoop/lib/native/libsnappy.so,
>> still no luck.
>>
>> What is going on here? Why won't Hadoop find my native libraries? Is
>> there any log where I can check what went wrong during loading?
>>
>> Best regards
>> Yves
>>
Re: Hadoop native and snappy library
Posted by Yves Weissig <we...@uni-mainz.de>.
Thanks... after looking at the sourcecode of the native library loading
process and activating the DEBUG messages I found this in the "hadoop.log":
2014-02-11 14:01:14,084 DEBUG org.apache.hadoop.util.NativeCodeLoader:
Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError:
/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0: /lib64/libc.so.6: version
`GLIBC_2.14' not found (required by
/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0)
Is it possible to compile the native library with glibc 2.12, because at
the moment I am not able to upgrade glibc on the servers which run Hadoop.
Best regards
Yves
On 11.02.2014 13:29, Ted Yu wrote:
> See https://issues.apache.org/jira/browse/HBASE-10303
>
> And https://hbase.apache.org/book.html#snappy.compression
>
> Cheers
>
> On Feb 11, 2014, at 4:16 AM, Yves Weissig <weissig@uni-mainz.de
> <ma...@uni-mainz.de>> wrote:
>
>> Hi list,
>>
>> I'm trying to enable the Hadoop native library and the snappy library
>> for compression in Hadoop 2.2.0, but I always end up with:
>>
>> ./hadoop/bin/hadoop checknative -a
>> Native library checking:
>> hadoop: false
>> zlib: false
>> snappy: false
>> lz4: false
>> bzip2: false
>>
>> I compiled hadoop-2.2.0-src from scratch for x64 and put the resulting
>> .so in hadoop/lib/native/. I also compiled snappy from scratch and put
>> it there. In a different approach I installed snappy via sudo apt-get
>> and then linked the resulting .so to hadoop/lib/native/libsnappy.so,
>> still no luck.
>>
>> What is going on here? Why won't Hadoop find my native libraries? Is
>> there any log where I can check what went wrong during loading?
>>
>> Best regards
>> Yves
>>
Re: Hadoop native and snappy library
Posted by Yves Weissig <we...@uni-mainz.de>.
Thanks... after looking at the sourcecode of the native library loading
process and activating the DEBUG messages I found this in the "hadoop.log":
2014-02-11 14:01:14,084 DEBUG org.apache.hadoop.util.NativeCodeLoader:
Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError:
/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0: /lib64/libc.so.6: version
`GLIBC_2.14' not found (required by
/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0)
Is it possible to compile the native library with glibc 2.12, because at
the moment I am not able to upgrade glibc on the servers which run Hadoop.
Best regards
Yves
On 11.02.2014 13:29, Ted Yu wrote:
> See https://issues.apache.org/jira/browse/HBASE-10303
>
> And https://hbase.apache.org/book.html#snappy.compression
>
> Cheers
>
> On Feb 11, 2014, at 4:16 AM, Yves Weissig <weissig@uni-mainz.de
> <ma...@uni-mainz.de>> wrote:
>
>> Hi list,
>>
>> I'm trying to enable the Hadoop native library and the snappy library
>> for compression in Hadoop 2.2.0, but I always end up with:
>>
>> ./hadoop/bin/hadoop checknative -a
>> Native library checking:
>> hadoop: false
>> zlib: false
>> snappy: false
>> lz4: false
>> bzip2: false
>>
>> I compiled hadoop-2.2.0-src from scratch for x64 and put the resulting
>> .so in hadoop/lib/native/. I also compiled snappy from scratch and put
>> it there. In a different approach I installed snappy via sudo apt-get
>> and then linked the resulting .so to hadoop/lib/native/libsnappy.so,
>> still no luck.
>>
>> What is going on here? Why won't Hadoop find my native libraries? Is
>> there any log where I can check what went wrong during loading?
>>
>> Best regards
>> Yves
>>
Re: Hadoop native and snappy library
Posted by Yves Weissig <we...@uni-mainz.de>.
Thanks... after looking at the sourcecode of the native library loading
process and activating the DEBUG messages I found this in the "hadoop.log":
2014-02-11 14:01:14,084 DEBUG org.apache.hadoop.util.NativeCodeLoader:
Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError:
/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0: /lib64/libc.so.6: version
`GLIBC_2.14' not found (required by
/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0)
Is it possible to compile the native library with glibc 2.12, because at
the moment I am not able to upgrade glibc on the servers which run Hadoop.
Best regards
Yves
On 11.02.2014 13:29, Ted Yu wrote:
> See https://issues.apache.org/jira/browse/HBASE-10303
>
> And https://hbase.apache.org/book.html#snappy.compression
>
> Cheers
>
> On Feb 11, 2014, at 4:16 AM, Yves Weissig <weissig@uni-mainz.de
> <ma...@uni-mainz.de>> wrote:
>
>> Hi list,
>>
>> I'm trying to enable the Hadoop native library and the snappy library
>> for compression in Hadoop 2.2.0, but I always end up with:
>>
>> ./hadoop/bin/hadoop checknative -a
>> Native library checking:
>> hadoop: false
>> zlib: false
>> snappy: false
>> lz4: false
>> bzip2: false
>>
>> I compiled hadoop-2.2.0-src from scratch for x64 and put the resulting
>> .so in hadoop/lib/native/. I also compiled snappy from scratch and put
>> it there. In a different approach I installed snappy via sudo apt-get
>> and then linked the resulting .so to hadoop/lib/native/libsnappy.so,
>> still no luck.
>>
>> What is going on here? Why won't Hadoop find my native libraries? Is
>> there any log where I can check what went wrong during loading?
>>
>> Best regards
>> Yves
>>
Re: Hadoop native and snappy library
Posted by Ted Yu <yu...@gmail.com>.
See https://issues.apache.org/jira/browse/HBASE-10303
And https://hbase.apache.org/book.html#snappy.compression
Cheers
On Feb 11, 2014, at 4:16 AM, Yves Weissig <we...@uni-mainz.de> wrote:
> Hi list,
>
> I'm trying to enable the Hadoop native library and the snappy library
> for compression in Hadoop 2.2.0, but I always end up with:
>
> ./hadoop/bin/hadoop checknative -a
> Native library checking:
> hadoop: false
> zlib: false
> snappy: false
> lz4: false
> bzip2: false
>
> I compiled hadoop-2.2.0-src from scratch for x64 and put the resulting
> .so in hadoop/lib/native/. I also compiled snappy from scratch and put
> it there. In a different approach I installed snappy via sudo apt-get
> and then linked the resulting .so to hadoop/lib/native/libsnappy.so,
> still no luck.
>
> What is going on here? Why won't Hadoop find my native libraries? Is
> there any log where I can check what went wrong during loading?
>
> Best regards
> Yves
>
Re: Hadoop native and snappy library
Posted by Ted Yu <yu...@gmail.com>.
See https://issues.apache.org/jira/browse/HBASE-10303
And https://hbase.apache.org/book.html#snappy.compression
Cheers
On Feb 11, 2014, at 4:16 AM, Yves Weissig <we...@uni-mainz.de> wrote:
> Hi list,
>
> I'm trying to enable the Hadoop native library and the snappy library
> for compression in Hadoop 2.2.0, but I always end up with:
>
> ./hadoop/bin/hadoop checknative -a
> Native library checking:
> hadoop: false
> zlib: false
> snappy: false
> lz4: false
> bzip2: false
>
> I compiled hadoop-2.2.0-src from scratch for x64 and put the resulting
> .so in hadoop/lib/native/. I also compiled snappy from scratch and put
> it there. In a different approach I installed snappy via sudo apt-get
> and then linked the resulting .so to hadoop/lib/native/libsnappy.so,
> still no luck.
>
> What is going on here? Why won't Hadoop find my native libraries? Is
> there any log where I can check what went wrong during loading?
>
> Best regards
> Yves
>
Re: Hadoop native and snappy library
Posted by Ted Yu <yu...@gmail.com>.
See https://issues.apache.org/jira/browse/HBASE-10303
And https://hbase.apache.org/book.html#snappy.compression
Cheers
On Feb 11, 2014, at 4:16 AM, Yves Weissig <we...@uni-mainz.de> wrote:
> Hi list,
>
> I'm trying to enable the Hadoop native library and the snappy library
> for compression in Hadoop 2.2.0, but I always end up with:
>
> ./hadoop/bin/hadoop checknative -a
> Native library checking:
> hadoop: false
> zlib: false
> snappy: false
> lz4: false
> bzip2: false
>
> I compiled hadoop-2.2.0-src from scratch for x64 and put the resulting
> .so in hadoop/lib/native/. I also compiled snappy from scratch and put
> it there. In a different approach I installed snappy via sudo apt-get
> and then linked the resulting .so to hadoop/lib/native/libsnappy.so,
> still no luck.
>
> What is going on here? Why won't Hadoop find my native libraries? Is
> there any log where I can check what went wrong during loading?
>
> Best regards
> Yves
>
Re: Hadoop native and snappy library
Posted by Ted Yu <yu...@gmail.com>.
See https://issues.apache.org/jira/browse/HBASE-10303
And https://hbase.apache.org/book.html#snappy.compression
Cheers
On Feb 11, 2014, at 4:16 AM, Yves Weissig <we...@uni-mainz.de> wrote:
> Hi list,
>
> I'm trying to enable the Hadoop native library and the snappy library
> for compression in Hadoop 2.2.0, but I always end up with:
>
> ./hadoop/bin/hadoop checknative -a
> Native library checking:
> hadoop: false
> zlib: false
> snappy: false
> lz4: false
> bzip2: false
>
> I compiled hadoop-2.2.0-src from scratch for x64 and put the resulting
> .so in hadoop/lib/native/. I also compiled snappy from scratch and put
> it there. In a different approach I installed snappy via sudo apt-get
> and then linked the resulting .so to hadoop/lib/native/libsnappy.so,
> still no luck.
>
> What is going on here? Why won't Hadoop find my native libraries? Is
> there any log where I can check what went wrong during loading?
>
> Best regards
> Yves
>