Re: About GBDT support

[Peter Higdon <pe...@gmail.com>]

Hi Jerry,

Does this mean that your modification of OpenMPI to run with on Hadoop is not
suitable to fit into the Mahout project? 

It may not be HPC, but it would offer versatile algorithm options while staying
in a familiar, de-facto standard, environment.

Thanks!

Peter

Re: About GBDT support

[Jerry Ye <je...@yahoo-inc.com>]

Hi Peter,
The MPI launcher I wrote essentially tricks Hadoop into thinking that there are mappers that run for a really long time but launches MPI jobs instead.  The main advantage is that I get to use existing Hadoop clusters to launch my MPI jobs.  The only way this would fit in with the Mahout project is to have Mahout launch these types of jobs and essentially act as a workflow manager.  Also, keep in mind that the MPI code would be in C/C++ as well.  Using RPC and implementing some of MPI's functionality in Mahout might be a better alternative.

My understanding is that there will be a MPI application manager in Hadoop YARN/NextGen.  So you don't have to wait too long if all you want to do is launch MPI jobs on an existing Hadoop cluster.

- jerry

On Sep 15, 2011, at 5:13 AM, Peter Higdon wrote:

> Hi Jerry,
> 
> Does this mean that your modification of OpenMPI to run with on Hadoop is not
> suitable to fit into the Mahout project? 
> 
> It may not be HPC, but it would offer versatile algorithm options while staying
> in a familiar, de-facto standard, environment.
> 
> Thanks!
> 
> Peter
>