You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Taeho Kang <tk...@gmail.com> on 2007/09/12 12:03:44 UTC
Running non-Java binaries on a "heterogenous" Hadoop cluster?
Hello all,
Has anyone been in a situation where hardwares in a hadoop cluster are
different when running non-Java binaries on Hadoop?
e.g. a c++ map-reduce binary was compiled on a 32-bit CPU, but Hadoop
cluster consists of both 32 and 64 bit machines.
My guess is that the result from this situation will be a big mess.
Does Hadoop have a system or rules to prevent or help such situations?
(e.g. a config system that prevents a 32 bit binary from running on a 64bit
node)
Also when compiling C++ map-reduce source codes with the given build.xml in
version 0.14.1, does it compile in 32 bit mode?
Any comments and ideas will be appreciated. Thank you in advance.
Regards,
Taeho
--
Taeho Kang [tkang.blogspot.com]
Re: Running non-Java binaries on a "heterogenous" Hadoop cluster?
Posted by Owen O'Malley <oo...@yahoo-inc.com>.
On Sep 12, 2007, at 4:07 AM, Arun C Murthy wrote:
>> Does Hadoop have a system or rules to prevent or help such
>> situations?
>> (e.g. a config system that prevents a 32 bit binary from running
>> on a 64bit
>> node)
>>
>
> No. Not as of today.
Such a framework would make sense for static clusters. We will likely
move more in the direction of using Torque or Condor to dynamically
allocate a cluster with the necessary hardware/OS for a given
application.
>> Also when compiling C++ map-reduce source codes with the given
>> build.xml in
>> version 0.14.1, does it compile in 32 bit mode?
I believe the autoconf file doesn't specify, so it is the default for
the platform. It would make sense to extend the build.xml to include
a variable to pass to the configure scripts when building c++.
-- Owen
Re: Running non-Java binaries on a "heterogenous" Hadoop cluster?
Posted by Arun C Murthy <ar...@yahoo-inc.com>.
On Wed, Sep 12, 2007 at 07:03:44PM +0900, Taeho Kang wrote:
>Hello all,
>
>Has anyone been in a situation where hardwares in a hadoop cluster are
>different when running non-Java binaries on Hadoop?
>e.g. a c++ map-reduce binary was compiled on a 32-bit CPU, but Hadoop
>cluster consists of both 32 and 64 bit machines.
>My guess is that the result from this situation will be a big mess.
>
Depends. Does your cluster have the same OS?
If so, I don't see a problem running a 32-bit linux executable on a 64-bit machine, but not vice-versa.
>Does Hadoop have a system or rules to prevent or help such situations?
>(e.g. a config system that prevents a 32 bit binary from running on a 64bit
>node)
>
No. Not as of today.
>Also when compiling C++ map-reduce source codes with the given build.xml in
>version 0.14.1, does it compile in 32 bit mode?
>
Not sure, I don't see it. I'll let Owen comment on this one...
Arun
>Any comments and ideas will be appreciated. Thank you in advance.
>
>Regards,
>Taeho
>
>--
>Taeho Kang [tkang.blogspot.com]