You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Steven Willis <sw...@compete.com> on 2012/08/27 20:11:03 UTC
RE: hadoop native libs 32 and 64 bit
(moving to common-dev)
Thanks Harsh,
So what's the final outcome of these changes? Do we get both 32 and 64 bit libraries in the release tarball? Will they be underneath an arch dir, or directly under lib/native? I'm just a bit confused because the issue you reference is:
"native libs should be under lib/native/ dir"
https://issues.apache.org/jira/browse/HADOOP-7874
but there's also this one:
"change location of the native libraries to lib instead of lib/native"
https://issues.apache.org/jira/browse/HADOOP-7996
-Steven Willis
> -----Original Message-----
> From: Harsh J [mailto:harsh@cloudera.com]
> Sent: Monday, August 27, 2012 1:17 PM
> To: user@hadoop.apache.org
> Subject: Re: hadoop native libs 32 and 64 bit
>
> Hi Steven,
>
> You may also use the common-dev@ lists for development
> discussions/issues around common elements :)
>
> Just for some context, this was changed in 2.x by us via:
> https://issues.apache.org/jira/browse/HADOOP-7874
>
> On Mon, Aug 27, 2012 at 10:39 PM, Steven Willis <sw...@compete.com>
> wrote:
> > Hi,
> >
> > I've been looking for both the 32 and 64 bit hadoop native libraries
> and it looks like the existence and location of these libraries keeps
> changing between releases. I downloaded the following releases:
> >
> > hadoop-0.22.0
> > hadoop-0.23.0
> > hadoop-0.23.1
> > hadoop-1.0.1
> > hadoop-1.0.2
> > hadoop-1.0.3
> > hadoop-2.0.0-alpha
> > hadoop-2.0.1-alpha
> >
> > and looked for the location of 'libhadoop.so*' and ran 'file' on any
> results, this is what I found:
> >
> > ./hadoop-0.22.0/lib/native/Linux-amd64-64/libhadoop.so: ELF 64-
> bit LSB shared object
> > ./hadoop-0.22.0/lib/native/Linux-amd64-64/libhadoop.so.1: ELF 64-
> bit LSB shared object
> > ./hadoop-0.22.0/lib/native/Linux-amd64-64/libhadoop.so.1.0.0: ELF 64-
> bit LSB shared object
> > ./hadoop-0.23.0/lib/libhadoop.so:
> symbolic link to `libhadoop.so.1.0.0'
> > ./hadoop-0.23.0/lib/libhadoop.so.1:
> symbolic link to `libhadoop.so.1.0.0'
> > ./hadoop-0.23.0/lib/libhadoop.so.1.0.0: ELF 32-
> bit LSB shared object
> > ./hadoop-0.23.1/lib/native/libhadoop.so:
> symbolic link to `libhadoop.so.1.0.0'
> > ./hadoop-0.23.1/lib/native/libhadoop.so.1:
> symbolic link to `libhadoop.so.1.0.0'
> > ./hadoop-0.23.1/lib/native/libhadoop.so.1.0.0: ELF 32-
> bit LSB shared object
> > ./hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so: ELF 64-
> bit LSB shared object
> > ./hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so.1: ELF 64-
> bit LSB shared object
> > ./hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so.1.0.0: ELF 64-
> bit LSB shared object
> > ./hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so: ELF 32-
> bit LSB shared object
> > ./hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so.1: ELF 32-
> bit LSB shared object
> > ./hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so.1.0.0: ELF 32-
> bit LSB shared object
> > ./hadoop-1.0.2/lib/native/Linux-amd64-64/libhadoop.so: ELF 64-
> bit LSB shared object
> > ./hadoop-1.0.2/lib/native/Linux-amd64-64/libhadoop.so.1: ELF 64-
> bit LSB shared object
> > ./hadoop-1.0.2/lib/native/Linux-amd64-64/libhadoop.so.1.0.0: ELF 64-
> bit LSB shared object
> > ./hadoop-1.0.2/lib/native/Linux-i386-32/libhadoop.so: ELF 32-
> bit LSB shared object
> > ./hadoop-1.0.2/lib/native/Linux-i386-32/libhadoop.so.1: ELF 32-
> bit LSB shared object
> > ./hadoop-1.0.2/lib/native/Linux-i386-32/libhadoop.so.1.0.0: ELF 32-
> bit LSB shared object
> > ./hadoop-1.0.3/lib/native/Linux-amd64-64/libhadoop.so: ELF 64-
> bit LSB shared object
> > ./hadoop-1.0.3/lib/native/Linux-amd64-64/libhadoop.so.1: ELF 64-
> bit LSB shared object
> > ./hadoop-1.0.3/lib/native/Linux-amd64-64/libhadoop.so.1.0.0: ELF 64-
> bit LSB shared object
> > ./hadoop-1.0.3/lib/native/Linux-i386-32/libhadoop.so: ELF 32-
> bit LSB shared object
> > ./hadoop-1.0.3/lib/native/Linux-i386-32/libhadoop.so.1: ELF 32-
> bit LSB shared object
> > ./hadoop-1.0.3/lib/native/Linux-i386-32/libhadoop.so.1.0.0: ELF 32-
> bit LSB shared object
> > ./hadoop-2.0.0-alpha/lib/native/libhadoop.so:
> symbolic link to `libhadoop.so.1.0.0'
> > ./hadoop-2.0.0-alpha/lib/native/libhadoop.so.1:
> symbolic link to `libhadoop.so.1.0.0'
> > ./hadoop-2.0.0-alpha/lib/native/libhadoop.so.1.0.0: ELF 32-
> bit LSB shared object
> > ./hadoop-2.0.1-alpha/lib/native/libhadoop.so:
> symbolic link to `libhadoop.so.1.0.0'
> > ./hadoop-2.0.1-alpha/lib/native/libhadoop.so.1:
> symbolic link to `libhadoop.so.1.0.0'
> > ./hadoop-2.0.1-alpha/lib/native/libhadoop.so.1.0.0: ELF 32-
> bit LSB shared object
> >
> > It looks like in 0.22.0, hadoop was only released with 64 bit libs.
> They were put in an arch directory under 'lib/native'. And there are
> copies for the .so files not symlinks. Hadoop 0.23.0 has only 32 bit
> libs which are directly under 'lib' (no 'native' or arch dir), and
> there are symlinks for the .so files. Hadoop 0.23.1 still has just 32
> bit libs and symlinks, but they're now under 'lib/native'. From 1.0.1
> to 1.0.3 we have both 32 and 64 bit libs and they're back under
> 'lib/native/ARCH', and we're back to having copies of the libs instead
> of symlinks. In 2.0.0-alpha and 2.0.1-alpha they go back to 32 bit only
> and under 'lib/native', and we get symlinks back.
> >
> > The following is a table of what we get with each release:
> >
> > release 32 64 archdir symlinks
> > hadoop-0.22.0 N Y Y N
> > hadoop-0.23.0 Y N N* Y
> > hadoop-0.23.1 Y N N Y
> > hadoop-1.0.1 Y Y Y N
> > hadoop-1.0.2 Y Y Y N
> > hadoop-1.0.3 Y Y Y N
> > hadoop-2.0.0-alpha Y N N Y
> > hadoop-2.0.1-alpha Y N N Y
> >
> > *libs not even under 'native'
> >
> > Can the build process be changed to include 32 and 64 bit libs, put
> them under 'lib/native/ARCH', and use symlinks for the .so an .so.1
> files? Alternatively, maybe the native libraries should be split out
> into separate downloads, so you'd have something like:
> >
> > hadoop-X.Y.Z.tar.gz
> > hadoop-native-X.Y.Z.i386.tar.gz
> > hadoop-native-X.Y.Z.x86_64.tar.gz
> >
> > Is this something I should open a JIRA for?
> >
> > -Steven Willis
>
>
>
> --
> Harsh J
Re: hadoop native libs 32 and 64 bit
Posted by Harsh J <ha...@cloudera.com>.
I don't think the latter's been explained or done. Besides, I think
the latter is related more to packaging code (RPMs, etc.) than source
tarballs.
We ship source tarballs in Apache Hadoop releases, and Apache Bigtop
ships/will ship packaged artifacts (going forward). We don't currently
guarantee the availability of properly working precompiled native
artifacts for any platform today, but we can make a note to state so.
For instance, some of the current natives such as snappy fail to work
on CentOS5 if compiled on CentOS6, etc., so there's nothing we can do
aside of shipping sources with commands on building your own natives.
Or better, rely on Apache Bigtop to provide proper packages/package
building scripts?
On Mon, Aug 27, 2012 at 11:41 PM, Steven Willis <sw...@compete.com> wrote:
> (moving to common-dev)
>
> Thanks Harsh,
>
> So what's the final outcome of these changes? Do we get both 32 and 64 bit libraries in the release tarball? Will they be underneath an arch dir, or directly under lib/native? I'm just a bit confused because the issue you reference is:
>
> "native libs should be under lib/native/ dir"
> https://issues.apache.org/jira/browse/HADOOP-7874
>
> but there's also this one:
>
> "change location of the native libraries to lib instead of lib/native"
> https://issues.apache.org/jira/browse/HADOOP-7996
>
> -Steven Willis
>
>> -----Original Message-----
>> From: Harsh J [mailto:harsh@cloudera.com]
>> Sent: Monday, August 27, 2012 1:17 PM
>> To: user@hadoop.apache.org
>> Subject: Re: hadoop native libs 32 and 64 bit
>>
>> Hi Steven,
>>
>> You may also use the common-dev@ lists for development
>> discussions/issues around common elements :)
>>
>> Just for some context, this was changed in 2.x by us via:
>> https://issues.apache.org/jira/browse/HADOOP-7874
>>
>> On Mon, Aug 27, 2012 at 10:39 PM, Steven Willis <sw...@compete.com>
>> wrote:
>> > Hi,
>> >
>> > I've been looking for both the 32 and 64 bit hadoop native libraries
>> and it looks like the existence and location of these libraries keeps
>> changing between releases. I downloaded the following releases:
>> >
>> > hadoop-0.22.0
>> > hadoop-0.23.0
>> > hadoop-0.23.1
>> > hadoop-1.0.1
>> > hadoop-1.0.2
>> > hadoop-1.0.3
>> > hadoop-2.0.0-alpha
>> > hadoop-2.0.1-alpha
>> >
>> > and looked for the location of 'libhadoop.so*' and ran 'file' on any
>> results, this is what I found:
>> >
>> > ./hadoop-0.22.0/lib/native/Linux-amd64-64/libhadoop.so: ELF 64-
>> bit LSB shared object
>> > ./hadoop-0.22.0/lib/native/Linux-amd64-64/libhadoop.so.1: ELF 64-
>> bit LSB shared object
>> > ./hadoop-0.22.0/lib/native/Linux-amd64-64/libhadoop.so.1.0.0: ELF 64-
>> bit LSB shared object
>> > ./hadoop-0.23.0/lib/libhadoop.so:
>> symbolic link to `libhadoop.so.1.0.0'
>> > ./hadoop-0.23.0/lib/libhadoop.so.1:
>> symbolic link to `libhadoop.so.1.0.0'
>> > ./hadoop-0.23.0/lib/libhadoop.so.1.0.0: ELF 32-
>> bit LSB shared object
>> > ./hadoop-0.23.1/lib/native/libhadoop.so:
>> symbolic link to `libhadoop.so.1.0.0'
>> > ./hadoop-0.23.1/lib/native/libhadoop.so.1:
>> symbolic link to `libhadoop.so.1.0.0'
>> > ./hadoop-0.23.1/lib/native/libhadoop.so.1.0.0: ELF 32-
>> bit LSB shared object
>> > ./hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so: ELF 64-
>> bit LSB shared object
>> > ./hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so.1: ELF 64-
>> bit LSB shared object
>> > ./hadoop-1.0.1/lib/native/Linux-amd64-64/libhadoop.so.1.0.0: ELF 64-
>> bit LSB shared object
>> > ./hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so: ELF 32-
>> bit LSB shared object
>> > ./hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so.1: ELF 32-
>> bit LSB shared object
>> > ./hadoop-1.0.1/lib/native/Linux-i386-32/libhadoop.so.1.0.0: ELF 32-
>> bit LSB shared object
>> > ./hadoop-1.0.2/lib/native/Linux-amd64-64/libhadoop.so: ELF 64-
>> bit LSB shared object
>> > ./hadoop-1.0.2/lib/native/Linux-amd64-64/libhadoop.so.1: ELF 64-
>> bit LSB shared object
>> > ./hadoop-1.0.2/lib/native/Linux-amd64-64/libhadoop.so.1.0.0: ELF 64-
>> bit LSB shared object
>> > ./hadoop-1.0.2/lib/native/Linux-i386-32/libhadoop.so: ELF 32-
>> bit LSB shared object
>> > ./hadoop-1.0.2/lib/native/Linux-i386-32/libhadoop.so.1: ELF 32-
>> bit LSB shared object
>> > ./hadoop-1.0.2/lib/native/Linux-i386-32/libhadoop.so.1.0.0: ELF 32-
>> bit LSB shared object
>> > ./hadoop-1.0.3/lib/native/Linux-amd64-64/libhadoop.so: ELF 64-
>> bit LSB shared object
>> > ./hadoop-1.0.3/lib/native/Linux-amd64-64/libhadoop.so.1: ELF 64-
>> bit LSB shared object
>> > ./hadoop-1.0.3/lib/native/Linux-amd64-64/libhadoop.so.1.0.0: ELF 64-
>> bit LSB shared object
>> > ./hadoop-1.0.3/lib/native/Linux-i386-32/libhadoop.so: ELF 32-
>> bit LSB shared object
>> > ./hadoop-1.0.3/lib/native/Linux-i386-32/libhadoop.so.1: ELF 32-
>> bit LSB shared object
>> > ./hadoop-1.0.3/lib/native/Linux-i386-32/libhadoop.so.1.0.0: ELF 32-
>> bit LSB shared object
>> > ./hadoop-2.0.0-alpha/lib/native/libhadoop.so:
>> symbolic link to `libhadoop.so.1.0.0'
>> > ./hadoop-2.0.0-alpha/lib/native/libhadoop.so.1:
>> symbolic link to `libhadoop.so.1.0.0'
>> > ./hadoop-2.0.0-alpha/lib/native/libhadoop.so.1.0.0: ELF 32-
>> bit LSB shared object
>> > ./hadoop-2.0.1-alpha/lib/native/libhadoop.so:
>> symbolic link to `libhadoop.so.1.0.0'
>> > ./hadoop-2.0.1-alpha/lib/native/libhadoop.so.1:
>> symbolic link to `libhadoop.so.1.0.0'
>> > ./hadoop-2.0.1-alpha/lib/native/libhadoop.so.1.0.0: ELF 32-
>> bit LSB shared object
>> >
>> > It looks like in 0.22.0, hadoop was only released with 64 bit libs.
>> They were put in an arch directory under 'lib/native'. And there are
>> copies for the .so files not symlinks. Hadoop 0.23.0 has only 32 bit
>> libs which are directly under 'lib' (no 'native' or arch dir), and
>> there are symlinks for the .so files. Hadoop 0.23.1 still has just 32
>> bit libs and symlinks, but they're now under 'lib/native'. From 1.0.1
>> to 1.0.3 we have both 32 and 64 bit libs and they're back under
>> 'lib/native/ARCH', and we're back to having copies of the libs instead
>> of symlinks. In 2.0.0-alpha and 2.0.1-alpha they go back to 32 bit only
>> and under 'lib/native', and we get symlinks back.
>> >
>> > The following is a table of what we get with each release:
>> >
>> > release 32 64 archdir symlinks
>> > hadoop-0.22.0 N Y Y N
>> > hadoop-0.23.0 Y N N* Y
>> > hadoop-0.23.1 Y N N Y
>> > hadoop-1.0.1 Y Y Y N
>> > hadoop-1.0.2 Y Y Y N
>> > hadoop-1.0.3 Y Y Y N
>> > hadoop-2.0.0-alpha Y N N Y
>> > hadoop-2.0.1-alpha Y N N Y
>> >
>> > *libs not even under 'native'
>> >
>> > Can the build process be changed to include 32 and 64 bit libs, put
>> them under 'lib/native/ARCH', and use symlinks for the .so an .so.1
>> files? Alternatively, maybe the native libraries should be split out
>> into separate downloads, so you'd have something like:
>> >
>> > hadoop-X.Y.Z.tar.gz
>> > hadoop-native-X.Y.Z.i386.tar.gz
>> > hadoop-native-X.Y.Z.x86_64.tar.gz
>> >
>> > Is this something I should open a JIRA for?
>> >
>> > -Steven Willis
>>
>>
>>
>> --
>> Harsh J
--
Harsh J