You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by "Pastrana, Rodrigo (RIS-BCT)" <Ro...@lexisnexis.com> on 2012/09/25 05:20:24 UTC
libhdfs install dep
Anybody know why libhdfs.so is not found by package managers on CentOS 64 and OpenSuse64?
I hava an rpm which declares Hadoop as a dependacy, but the package managers (KPackageKit, zypper, etc) report libhdfs.so as a missing dependency eventhough Hadoop has been installed via rpm package, and libhdfs.so is installed as well.
Thanks, Rodrigo.
Re: libhdfs install dep
Posted by Brian Bockelman <bb...@cse.unl.edu>.
Hi Rodrigo,
The hadoop RPMs are a bit deficient compared to those you would find from your Linux distribution.
For example, look at the Apache RPM you used:
[bbockelm@rcf-bockelman ~]$ rpm -qp http://mirrors.sonic.net/apache/hadoop/common/hadoop-1.0.3/hadoop-1.0.3-1.x86_64.rpm --provides
hadoop
hadoop = 1.0.3-1
Normally, you would expect to see something like this (using the CDH4 distribution as an example) as it contains a shared library:
[bbockelm@brian-test ~]$ rpm -q --provides hadoop-libhdfs
libhdfs.so.0()(64bit)
hadoop-libhdfs = 2.0.0+88-1.cdh4.0.0.p0.30.osg.el5
libhdfs.so.0
hadoop-libhdfs = 2.0.0+88-1.cdh4.0.0.p0.30.osg.el5
Because the Apache RPM does not list itself as providing libhdfs.so.0()(64bit), it breaks your automatic RPM dependency detection.
[Aside: I know from experience that building a high-quality (as in, follows the Fedora Packaging Guidelines) RPM for Java software is incredibly hard as the packaging approaches between the Linux distributions and Java community are incredibly divergent. Not to say that the Java approach is inherently wrong, it's just different, and does not translate naturally to RPM. Accordingly, to take Hadoop and make a rule-abiding RPM in Fedora would be hundreds of hours of work. It's one of those things that appear to be much easier than it is to accomplish.]
The Hadoop community is very friendly, and I'm sure they would accept any patches to fix this oversight in future releases.
Brian
On Sep 25, 2012, at 7:57 AM, "Pastrana, Rodrigo (RIS-BCT)" <Ro...@lexisnexis.com> wrote:
> Leo, yes I'm working with hadoop-1.0.1-1.amd64.rpm from Apache's download site.
> The rpm installs libhdfs in /usr/lib64 so I'm not sure why I would need the hadoop-<*>libhdfs* rpm.
>
> Any idea why the installed /usr/lib64/libhdfs.so is not detected by the package managers?
>
> Thanks, Rodrigo.
>
> -----Original Message-----
> From: Leo Leung [mailto:lleung@ddn.com]
> Sent: Tuesday, September 25, 2012 2:11 AM
> To: common-user@hadoop.apache.org
> Subject: RE: libhdfs install dep
>
> Rodrigo,
> Assuming you are asking for hadoop 1.x
>
> You are missing the hadoop-<*>libhdfs* rpm.
> Build it or get it from the vendor you got your hadoop from.
>
>
>
> -----Original Message-----
> From: Pastrana, Rodrigo (RIS-BCT) [mailto:Rodrigo.Pastrana@lexisnexis.com]
> Sent: Monday, September 24, 2012 8:20 PM
> To: 'core-user@hadoop.apache.org'
> Subject: libhdfs install dep
>
> Anybody know why libhdfs.so is not found by package managers on CentOS 64 and OpenSuse64?
>
> I hava an rpm which declares Hadoop as a dependacy, but the package managers (KPackageKit, zypper, etc) report libhdfs.so as a missing dependency eventhough Hadoop has been installed via rpm package, and libhdfs.so is installed as well.
>
> Thanks, Rodrigo.
>
>
> -----------------------------------------
> The information contained in this e-mail message is intended only
> for the personal and confidential use of the recipient(s) named
> above. This message may be an attorney-client communication and/or
> work product and as such is privileged and confidential. If the
> reader of this message is not the intended recipient or an agent
> responsible for delivering it to the intended recipient, you are
> hereby notified that you have received this document in error and
> that any review, dissemination, distribution, or copying of this
> message is strictly prohibited. If you have received this
> communication in error, please notify us immediately by e-mail, and
> delete the original message.
RE: libhdfs install dep
Posted by Leo Leung <ll...@ddn.com>.
I see,
The apache RPM distro does include the libhdfs* binaries. (And it is self-contained)
For the Apache Distro (once installed) Yum,zipper,KPack* only knows about hadoop-1.0.1-1.amd* as the provider for the libhdfs
(check with rpm -qlp hadoop<version>.rpm -- this will give you all of the files it has/installs or yum deplist <package>)
So the package that you are trying to install, which is failing the dependency, has a requirement for
something name "hadoop" (per your e-mail) as a package that contains the libhdfs. (This is not what hadoop-1.0.1-1 claims)
To solve this: The brute's way
Force the install of the rpm without dependencies.
rpm -i --nodeps <the-package>.rpm
Another way.
Match up all your RPM package dependencies from the vendor(s)
1) Talk to the <the-package>.rpm provider, to see if they have the proper packaging for the apache RPM distro.
2) Use or find a hadoop distro from [a] vendor that has all the proper "package" naming resolution.
Good luck
-----Original Message-----
From: Pastrana, Rodrigo (RIS-BCT) [mailto:Rodrigo.Pastrana@lexisnexis.com]
Sent: Tuesday, September 25, 2012 5:58 AM
To: common-user@hadoop.apache.org
Subject: RE: libhdfs install dep
Leo, yes I'm working with hadoop-1.0.1-1.amd64.rpm from Apache's download site.
The rpm installs libhdfs in /usr/lib64 so I'm not sure why I would need the hadoop-<*>libhdfs* rpm.
Any idea why the installed /usr/lib64/libhdfs.so is not detected by the package managers?
Thanks, Rodrigo.
-----Original Message-----
From: Leo Leung [mailto:lleung@ddn.com]
Sent: Tuesday, September 25, 2012 2:11 AM
To: common-user@hadoop.apache.org
Subject: RE: libhdfs install dep
Rodrigo,
Assuming you are asking for hadoop 1.x
You are missing the hadoop-<*>libhdfs* rpm.
Build it or get it from the vendor you got your hadoop from.
-----Original Message-----
From: Pastrana, Rodrigo (RIS-BCT) [mailto:Rodrigo.Pastrana@lexisnexis.com]
Sent: Monday, September 24, 2012 8:20 PM
To: 'core-user@hadoop.apache.org'
Subject: libhdfs install dep
Anybody know why libhdfs.so is not found by package managers on CentOS 64 and OpenSuse64?
I hava an rpm which declares Hadoop as a dependacy, but the package managers (KPackageKit, zypper, etc) report libhdfs.so as a missing dependency eventhough Hadoop has been installed via rpm package, and libhdfs.so is installed as well.
Thanks, Rodrigo.
-----------------------------------------
The information contained in this e-mail message is intended only for the personal and confidential use of the recipient(s) named above. This message may be an attorney-client communication and/or work product and as such is privileged and confidential. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail, and delete the original message.
Re: libhdfs install dep
Posted by Harsh J <ha...@cloudera.com>.
I'd recommend using the packages for Apache Hadoop from Apache Bigtop
(https://cwiki.apache.org/confluence/display/BIGTOP). The ones
upstream (here) aren't maintained as much these days.
On Tue, Sep 25, 2012 at 6:27 PM, Pastrana, Rodrigo (RIS-BCT)
<Ro...@lexisnexis.com> wrote:
> Leo, yes I'm working with hadoop-1.0.1-1.amd64.rpm from Apache's download site.
> The rpm installs libhdfs in /usr/lib64 so I'm not sure why I would need the hadoop-<*>libhdfs* rpm.
>
> Any idea why the installed /usr/lib64/libhdfs.so is not detected by the package managers?
>
> Thanks, Rodrigo.
>
> -----Original Message-----
> From: Leo Leung [mailto:lleung@ddn.com]
> Sent: Tuesday, September 25, 2012 2:11 AM
> To: common-user@hadoop.apache.org
> Subject: RE: libhdfs install dep
>
> Rodrigo,
> Assuming you are asking for hadoop 1.x
>
> You are missing the hadoop-<*>libhdfs* rpm.
> Build it or get it from the vendor you got your hadoop from.
>
>
>
> -----Original Message-----
> From: Pastrana, Rodrigo (RIS-BCT) [mailto:Rodrigo.Pastrana@lexisnexis.com]
> Sent: Monday, September 24, 2012 8:20 PM
> To: 'core-user@hadoop.apache.org'
> Subject: libhdfs install dep
>
> Anybody know why libhdfs.so is not found by package managers on CentOS 64 and OpenSuse64?
>
> I hava an rpm which declares Hadoop as a dependacy, but the package managers (KPackageKit, zypper, etc) report libhdfs.so as a missing dependency eventhough Hadoop has been installed via rpm package, and libhdfs.so is installed as well.
>
> Thanks, Rodrigo.
>
>
> -----------------------------------------
> The information contained in this e-mail message is intended only
> for the personal and confidential use of the recipient(s) named
> above. This message may be an attorney-client communication and/or
> work product and as such is privileged and confidential. If the
> reader of this message is not the intended recipient or an agent
> responsible for delivering it to the intended recipient, you are
> hereby notified that you have received this document in error and
> that any review, dissemination, distribution, or copying of this
> message is strictly prohibited. If you have received this
> communication in error, please notify us immediately by e-mail, and
> delete the original message.
--
Harsh J
RE: libhdfs install dep
Posted by "Pastrana, Rodrigo (RIS-BCT)" <Ro...@lexisnexis.com>.
Leo, yes I'm working with hadoop-1.0.1-1.amd64.rpm from Apache's download site.
The rpm installs libhdfs in /usr/lib64 so I'm not sure why I would need the hadoop-<*>libhdfs* rpm.
Any idea why the installed /usr/lib64/libhdfs.so is not detected by the package managers?
Thanks, Rodrigo.
-----Original Message-----
From: Leo Leung [mailto:lleung@ddn.com]
Sent: Tuesday, September 25, 2012 2:11 AM
To: common-user@hadoop.apache.org
Subject: RE: libhdfs install dep
Rodrigo,
Assuming you are asking for hadoop 1.x
You are missing the hadoop-<*>libhdfs* rpm.
Build it or get it from the vendor you got your hadoop from.
-----Original Message-----
From: Pastrana, Rodrigo (RIS-BCT) [mailto:Rodrigo.Pastrana@lexisnexis.com]
Sent: Monday, September 24, 2012 8:20 PM
To: 'core-user@hadoop.apache.org'
Subject: libhdfs install dep
Anybody know why libhdfs.so is not found by package managers on CentOS 64 and OpenSuse64?
I hava an rpm which declares Hadoop as a dependacy, but the package managers (KPackageKit, zypper, etc) report libhdfs.so as a missing dependency eventhough Hadoop has been installed via rpm package, and libhdfs.so is installed as well.
Thanks, Rodrigo.
-----------------------------------------
The information contained in this e-mail message is intended only
for the personal and confidential use of the recipient(s) named
above. This message may be an attorney-client communication and/or
work product and as such is privileged and confidential. If the
reader of this message is not the intended recipient or an agent
responsible for delivering it to the intended recipient, you are
hereby notified that you have received this document in error and
that any review, dissemination, distribution, or copying of this
message is strictly prohibited. If you have received this
communication in error, please notify us immediately by e-mail, and
delete the original message.
RE: libhdfs install dep
Posted by Leo Leung <ll...@ddn.com>.
Rodrigo,
Assuming you are asking for hadoop 1.x
You are missing the hadoop-<*>libhdfs* rpm.
Build it or get it from the vendor you got your hadoop from.
-----Original Message-----
From: Pastrana, Rodrigo (RIS-BCT) [mailto:Rodrigo.Pastrana@lexisnexis.com]
Sent: Monday, September 24, 2012 8:20 PM
To: 'core-user@hadoop.apache.org'
Subject: libhdfs install dep
Anybody know why libhdfs.so is not found by package managers on CentOS 64 and OpenSuse64?
I hava an rpm which declares Hadoop as a dependacy, but the package managers (KPackageKit, zypper, etc) report libhdfs.so as a missing dependency eventhough Hadoop has been installed via rpm package, and libhdfs.so is installed as well.
Thanks, Rodrigo.