You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@hadoop.apache.org by sam liu <sa...@gmail.com> on 2013/06/30 14:32:59 UTC

How to write/run MPI program on Yarn?

Hi Experts,

Does Hadoop 2.0.5-alpha supports MPI programs?

- If yes, is there any example of writing a MPI program on Yarn? How to
write the client side code to configure and submit the MPI job?
- If no, which Hadoop version will support MPI programming?

Thanks!

Re: How to write/run MPI program on Yarn?

Posted by Olivier Renault <or...@hortonworks.com>.

Sam,

The fundamental idea of YARN is to split up the two major functionalities of
the JobTracker, resource management and job scheduling/monitoring, into
separate daemons. With YARN, we're going to be able to run multiple
workload, one of them being MapReduce another one being MPI. So mpich2-yarn
is the port of MPI to run on top of yarn.

You may want to read :
http://hortonworks.com/hadoop/yarn/
http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html

Hope it helps,
Olivier


On 2 July 2013 02:21, sam liu <sa...@gmail.com> wrote:

> Any one could help answer above questions? Thanks a lot!
>
>
> 2013/7/1 sam liu <sa...@gmail.com>
>
>> Thanks Pramod and Clark!
>>
>> 1. What's the relationship of Hadoop 2.x branch and mpich2-yarn project?
>> 2. Does Hadoop 2.x branch plan to include MPI implementation? I mentioned
>> there is already a JIRA:
>> https://issues.apache.org/jira/browse/MAPREDUCE-2911
>>
>>
>> 2013/7/1 Clark Yang (杨卓荦) <ya...@gmail.com>
>>
>> Hi, sam
>>>
>>>    Please try it following the README.md on
>>> https://github.com/clarkyzl/mpich2-yarn/.
>>>   The design is simple ,make each nodemanger runs a "smpd" daemon and
>>> make application master run "mpiexec" to submit mpi jobs. The biggest
>>> problem here, I think, is the yarn scheduler do not support gang
>>> secheduling algorithm, and the scheduling is very very slow.
>>>
>>>   Any ideas please feel free to share.
>>>
>>> Cheers,
>>> Zhuoluo (Clark) Yang
>>>
>>>
>>> 2013/6/30 Pramod N <np...@gmail.com>
>>>
>>>> Hi Sam,
>>>> This might interest you.
>>>> https://github.com/clarkyzl/mpich2-yarn
>>>>
>>>>
>>>>
>>>> Pramod N <http://atmachinelearner.blogspot.in>
>>>> Bruce Wayne of web
>>>> @machinelearner <https://twitter.com/machinelearner>
>>>>
>>>> --
>>>>
>>>>
>>>> On Sun, Jun 30, 2013 at 6:02 PM, sam liu <sa...@gmail.com>wrote:
>>>>
>>>>> Hi Experts,
>>>>>
>>>>> Does Hadoop 2.0.5-alpha supports MPI programs?
>>>>>
>>>>> - If yes, is there any example of writing a MPI program on Yarn? How
>>>>> to write the client side code to configure and submit the MPI job?
>>>>> - If no, which Hadoop version will support MPI programming?
>>>>>
>>>>> Thanks!
>>>>>
>>>>
>>>>
>>>
>>
>


-- 
Olivier Renault
Solution Engineer - Big Data - Hortonworks, Inc.
+44 7500 933 036
orenault@hortonworks.com
www.hortonworks.com
<http://hortonworks.com/products/hortonworks-sandbox/>

Re: How to write/run MPI program on Yarn?

Posted by Olivier Renault <or...@hortonworks.com>.

Sam,

The fundamental idea of YARN is to split up the two major functionalities of
the JobTracker, resource management and job scheduling/monitoring, into
separate daemons. With YARN, we're going to be able to run multiple
workload, one of them being MapReduce another one being MPI. So mpich2-yarn
is the port of MPI to run on top of yarn.

You may want to read :
http://hortonworks.com/hadoop/yarn/
http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html

Hope it helps,
Olivier


On 2 July 2013 02:21, sam liu <sa...@gmail.com> wrote:

> Any one could help answer above questions? Thanks a lot!
>
>
> 2013/7/1 sam liu <sa...@gmail.com>
>
>> Thanks Pramod and Clark!
>>
>> 1. What's the relationship of Hadoop 2.x branch and mpich2-yarn project?
>> 2. Does Hadoop 2.x branch plan to include MPI implementation? I mentioned
>> there is already a JIRA:
>> https://issues.apache.org/jira/browse/MAPREDUCE-2911
>>
>>
>> 2013/7/1 Clark Yang (杨卓荦) <ya...@gmail.com>
>>
>> Hi, sam
>>>
>>>    Please try it following the README.md on
>>> https://github.com/clarkyzl/mpich2-yarn/.
>>>   The design is simple ,make each nodemanger runs a "smpd" daemon and
>>> make application master run "mpiexec" to submit mpi jobs. The biggest
>>> problem here, I think, is the yarn scheduler do not support gang
>>> secheduling algorithm, and the scheduling is very very slow.
>>>
>>>   Any ideas please feel free to share.
>>>
>>> Cheers,
>>> Zhuoluo (Clark) Yang
>>>
>>>
>>> 2013/6/30 Pramod N <np...@gmail.com>
>>>
>>>> Hi Sam,
>>>> This might interest you.
>>>> https://github.com/clarkyzl/mpich2-yarn
>>>>
>>>>
>>>>
>>>> Pramod N <http://atmachinelearner.blogspot.in>
>>>> Bruce Wayne of web
>>>> @machinelearner <https://twitter.com/machinelearner>
>>>>
>>>> --
>>>>
>>>>
>>>> On Sun, Jun 30, 2013 at 6:02 PM, sam liu <sa...@gmail.com>wrote:
>>>>
>>>>> Hi Experts,
>>>>>
>>>>> Does Hadoop 2.0.5-alpha supports MPI programs?
>>>>>
>>>>> - If yes, is there any example of writing a MPI program on Yarn? How
>>>>> to write the client side code to configure and submit the MPI job?
>>>>> - If no, which Hadoop version will support MPI programming?
>>>>>
>>>>> Thanks!
>>>>>
>>>>
>>>>
>>>
>>
>


-- 
Olivier Renault
Solution Engineer - Big Data - Hortonworks, Inc.
+44 7500 933 036
orenault@hortonworks.com
www.hortonworks.com
<http://hortonworks.com/products/hortonworks-sandbox/>

Re: How to write/run MPI program on Yarn?

Posted by Olivier Renault <or...@hortonworks.com>.

Sam,

The fundamental idea of YARN is to split up the two major functionalities of
the JobTracker, resource management and job scheduling/monitoring, into
separate daemons. With YARN, we're going to be able to run multiple
workload, one of them being MapReduce another one being MPI. So mpich2-yarn
is the port of MPI to run on top of yarn.

You may want to read :
http://hortonworks.com/hadoop/yarn/
http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html

Hope it helps,
Olivier


On 2 July 2013 02:21, sam liu <sa...@gmail.com> wrote:

> Any one could help answer above questions? Thanks a lot!
>
>
> 2013/7/1 sam liu <sa...@gmail.com>
>
>> Thanks Pramod and Clark!
>>
>> 1. What's the relationship of Hadoop 2.x branch and mpich2-yarn project?
>> 2. Does Hadoop 2.x branch plan to include MPI implementation? I mentioned
>> there is already a JIRA:
>> https://issues.apache.org/jira/browse/MAPREDUCE-2911
>>
>>
>> 2013/7/1 Clark Yang (杨卓荦) <ya...@gmail.com>
>>
>> Hi, sam
>>>
>>>    Please try it following the README.md on
>>> https://github.com/clarkyzl/mpich2-yarn/.
>>>   The design is simple ,make each nodemanger runs a "smpd" daemon and
>>> make application master run "mpiexec" to submit mpi jobs. The biggest
>>> problem here, I think, is the yarn scheduler do not support gang
>>> secheduling algorithm, and the scheduling is very very slow.
>>>
>>>   Any ideas please feel free to share.
>>>
>>> Cheers,
>>> Zhuoluo (Clark) Yang
>>>
>>>
>>> 2013/6/30 Pramod N <np...@gmail.com>
>>>
>>>> Hi Sam,
>>>> This might interest you.
>>>> https://github.com/clarkyzl/mpich2-yarn
>>>>
>>>>
>>>>
>>>> Pramod N <http://atmachinelearner.blogspot.in>
>>>> Bruce Wayne of web
>>>> @machinelearner <https://twitter.com/machinelearner>
>>>>
>>>> --
>>>>
>>>>
>>>> On Sun, Jun 30, 2013 at 6:02 PM, sam liu <sa...@gmail.com>wrote:
>>>>
>>>>> Hi Experts,
>>>>>
>>>>> Does Hadoop 2.0.5-alpha supports MPI programs?
>>>>>
>>>>> - If yes, is there any example of writing a MPI program on Yarn? How
>>>>> to write the client side code to configure and submit the MPI job?
>>>>> - If no, which Hadoop version will support MPI programming?
>>>>>
>>>>> Thanks!
>>>>>
>>>>
>>>>
>>>
>>
>


-- 
Olivier Renault
Solution Engineer - Big Data - Hortonworks, Inc.
+44 7500 933 036
orenault@hortonworks.com
www.hortonworks.com
<http://hortonworks.com/products/hortonworks-sandbox/>