You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by sam liu <sa...@gmail.com> on 2013/06/30 14:32:59 UTC
How to write/run MPI program on Yarn?
Hi Experts,
Does Hadoop 2.0.5-alpha supports MPI programs?
- If yes, is there any example of writing a MPI program on Yarn? How to
write the client side code to configure and submit the MPI job?
- If no, which Hadoop version will support MPI programming?
Thanks!
Re: How to write/run MPI program on Yarn?
Posted by Olivier Renault <or...@hortonworks.com>.
Sam,
The fundamental idea of YARN is to split up the two major functionalities of
the JobTracker, resource management and job scheduling/monitoring, into
separate daemons. With YARN, we're going to be able to run multiple
workload, one of them being MapReduce another one being MPI. So mpich2-yarn
is the port of MPI to run on top of yarn.
You may want to read :
http://hortonworks.com/hadoop/yarn/
http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html
Hope it helps,
Olivier
On 2 July 2013 02:21, sam liu <sa...@gmail.com> wrote:
> Any one could help answer above questions? Thanks a lot!
>
>
> 2013/7/1 sam liu <sa...@gmail.com>
>
>> Thanks Pramod and Clark!
>>
>> 1. What's the relationship of Hadoop 2.x branch and mpich2-yarn project?
>> 2. Does Hadoop 2.x branch plan to include MPI implementation? I mentioned
>> there is already a JIRA:
>> https://issues.apache.org/jira/browse/MAPREDUCE-2911
>>
>>
>> 2013/7/1 Clark Yang (杨卓荦) <ya...@gmail.com>
>>
>> Hi, sam
>>>
>>> Please try it following the README.md on
>>> https://github.com/clarkyzl/mpich2-yarn/.
>>> The design is simple ,make each nodemanger runs a "smpd" daemon and
>>> make application master run "mpiexec" to submit mpi jobs. The biggest
>>> problem here, I think, is the yarn scheduler do not support gang
>>> secheduling algorithm, and the scheduling is very very slow.
>>>
>>> Any ideas please feel free to share.
>>>
>>> Cheers,
>>> Zhuoluo (Clark) Yang
>>>
>>>
>>> 2013/6/30 Pramod N <np...@gmail.com>
>>>
>>>> Hi Sam,
>>>> This might interest you.
>>>> https://github.com/clarkyzl/mpich2-yarn
>>>>
>>>>
>>>>
>>>> Pramod N <http://atmachinelearner.blogspot.in>
>>>> Bruce Wayne of web
>>>> @machinelearner <https://twitter.com/machinelearner>
>>>>
>>>> --
>>>>
>>>>
>>>> On Sun, Jun 30, 2013 at 6:02 PM, sam liu <sa...@gmail.com>wrote:
>>>>
>>>>> Hi Experts,
>>>>>
>>>>> Does Hadoop 2.0.5-alpha supports MPI programs?
>>>>>
>>>>> - If yes, is there any example of writing a MPI program on Yarn? How
>>>>> to write the client side code to configure and submit the MPI job?
>>>>> - If no, which Hadoop version will support MPI programming?
>>>>>
>>>>> Thanks!
>>>>>
>>>>
>>>>
>>>
>>
>
--
Olivier Renault
Solution Engineer - Big Data - Hortonworks, Inc.
+44 7500 933 036
orenault@hortonworks.com
www.hortonworks.com
<http://hortonworks.com/products/hortonworks-sandbox/>
Re: How to write/run MPI program on Yarn?
Posted by Olivier Renault <or...@hortonworks.com>.
Sam,
The fundamental idea of YARN is to split up the two major functionalities of
the JobTracker, resource management and job scheduling/monitoring, into
separate daemons. With YARN, we're going to be able to run multiple
workload, one of them being MapReduce another one being MPI. So mpich2-yarn
is the port of MPI to run on top of yarn.
You may want to read :
http://hortonworks.com/hadoop/yarn/
http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html
Hope it helps,
Olivier
On 2 July 2013 02:21, sam liu <sa...@gmail.com> wrote:
> Any one could help answer above questions? Thanks a lot!
>
>
> 2013/7/1 sam liu <sa...@gmail.com>
>
>> Thanks Pramod and Clark!
>>
>> 1. What's the relationship of Hadoop 2.x branch and mpich2-yarn project?
>> 2. Does Hadoop 2.x branch plan to include MPI implementation? I mentioned
>> there is already a JIRA:
>> https://issues.apache.org/jira/browse/MAPREDUCE-2911
>>
>>
>> 2013/7/1 Clark Yang (杨卓荦) <ya...@gmail.com>
>>
>> Hi, sam
>>>
>>> Please try it following the README.md on
>>> https://github.com/clarkyzl/mpich2-yarn/.
>>> The design is simple ,make each nodemanger runs a "smpd" daemon and
>>> make application master run "mpiexec" to submit mpi jobs. The biggest
>>> problem here, I think, is the yarn scheduler do not support gang
>>> secheduling algorithm, and the scheduling is very very slow.
>>>
>>> Any ideas please feel free to share.
>>>
>>> Cheers,
>>> Zhuoluo (Clark) Yang
>>>
>>>
>>> 2013/6/30 Pramod N <np...@gmail.com>
>>>
>>>> Hi Sam,
>>>> This might interest you.
>>>> https://github.com/clarkyzl/mpich2-yarn
>>>>
>>>>
>>>>
>>>> Pramod N <http://atmachinelearner.blogspot.in>
>>>> Bruce Wayne of web
>>>> @machinelearner <https://twitter.com/machinelearner>
>>>>
>>>> --
>>>>
>>>>
>>>> On Sun, Jun 30, 2013 at 6:02 PM, sam liu <sa...@gmail.com>wrote:
>>>>
>>>>> Hi Experts,
>>>>>
>>>>> Does Hadoop 2.0.5-alpha supports MPI programs?
>>>>>
>>>>> - If yes, is there any example of writing a MPI program on Yarn? How
>>>>> to write the client side code to configure and submit the MPI job?
>>>>> - If no, which Hadoop version will support MPI programming?
>>>>>
>>>>> Thanks!
>>>>>
>>>>
>>>>
>>>
>>
>
--
Olivier Renault
Solution Engineer - Big Data - Hortonworks, Inc.
+44 7500 933 036
orenault@hortonworks.com
www.hortonworks.com
<http://hortonworks.com/products/hortonworks-sandbox/>
Re: How to write/run MPI program on Yarn?
Posted by Olivier Renault <or...@hortonworks.com>.
Sam,
The fundamental idea of YARN is to split up the two major functionalities of
the JobTracker, resource management and job scheduling/monitoring, into
separate daemons. With YARN, we're going to be able to run multiple
workload, one of them being MapReduce another one being MPI. So mpich2-yarn
is the port of MPI to run on top of yarn.
You may want to read :
http://hortonworks.com/hadoop/yarn/
http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html
Hope it helps,
Olivier
On 2 July 2013 02:21, sam liu <sa...@gmail.com> wrote:
> Any one could help answer above questions? Thanks a lot!
>
>
> 2013/7/1 sam liu <sa...@gmail.com>
>
>> Thanks Pramod and Clark!
>>
>> 1. What's the relationship of Hadoop 2.x branch and mpich2-yarn project?
>> 2. Does Hadoop 2.x branch plan to include MPI implementation? I mentioned
>> there is already a JIRA:
>> https://issues.apache.org/jira/browse/MAPREDUCE-2911
>>
>>
>> 2013/7/1 Clark Yang (杨卓荦) <ya...@gmail.com>
>>
>> Hi, sam
>>>
>>> Please try it following the README.md on
>>> https://github.com/clarkyzl/mpich2-yarn/.
>>> The design is simple ,make each nodemanger runs a "smpd" daemon and
>>> make application master run "mpiexec" to submit mpi jobs. The biggest
>>> problem here, I think, is the yarn scheduler do not support gang
>>> secheduling algorithm, and the scheduling is very very slow.
>>>
>>> Any ideas please feel free to share.
>>>
>>> Cheers,
>>> Zhuoluo (Clark) Yang
>>>
>>>
>>> 2013/6/30 Pramod N <np...@gmail.com>
>>>
>>>> Hi Sam,
>>>> This might interest you.
>>>> https://github.com/clarkyzl/mpich2-yarn
>>>>
>>>>
>>>>
>>>> Pramod N <http://atmachinelearner.blogspot.in>
>>>> Bruce Wayne of web
>>>> @machinelearner <https://twitter.com/machinelearner>
>>>>
>>>> --
>>>>
>>>>
>>>> On Sun, Jun 30, 2013 at 6:02 PM, sam liu <sa...@gmail.com>wrote:
>>>>
>>>>> Hi Experts,
>>>>>
>>>>> Does Hadoop 2.0.5-alpha supports MPI programs?
>>>>>
>>>>> - If yes, is there any example of writing a MPI program on Yarn? How
>>>>> to write the client side code to configure and submit the MPI job?
>>>>> - If no, which Hadoop version will support MPI programming?
>>>>>
>>>>> Thanks!
>>>>>
>>>>
>>>>
>>>
>>
>
--
Olivier Renault
Solution Engineer - Big Data - Hortonworks, Inc.
+44 7500 933 036
orenault@hortonworks.com
www.hortonworks.com
<http://hortonworks.com/products/hortonworks-sandbox/>
Re: How to write/run MPI program on Yarn?
Posted by Olivier Renault <or...@hortonworks.com>.
Sam,
The fundamental idea of YARN is to split up the two major functionalities of
the JobTracker, resource management and job scheduling/monitoring, into
separate daemons. With YARN, we're going to be able to run multiple
workload, one of them being MapReduce another one being MPI. So mpich2-yarn
is the port of MPI to run on top of yarn.
You may want to read :
http://hortonworks.com/hadoop/yarn/
http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html
Hope it helps,
Olivier
On 2 July 2013 02:21, sam liu <sa...@gmail.com> wrote:
> Any one could help answer above questions? Thanks a lot!
>
>
> 2013/7/1 sam liu <sa...@gmail.com>
>
>> Thanks Pramod and Clark!
>>
>> 1. What's the relationship of Hadoop 2.x branch and mpich2-yarn project?
>> 2. Does Hadoop 2.x branch plan to include MPI implementation? I mentioned
>> there is already a JIRA:
>> https://issues.apache.org/jira/browse/MAPREDUCE-2911
>>
>>
>> 2013/7/1 Clark Yang (杨卓荦) <ya...@gmail.com>
>>
>> Hi, sam
>>>
>>> Please try it following the README.md on
>>> https://github.com/clarkyzl/mpich2-yarn/.
>>> The design is simple ,make each nodemanger runs a "smpd" daemon and
>>> make application master run "mpiexec" to submit mpi jobs. The biggest
>>> problem here, I think, is the yarn scheduler do not support gang
>>> secheduling algorithm, and the scheduling is very very slow.
>>>
>>> Any ideas please feel free to share.
>>>
>>> Cheers,
>>> Zhuoluo (Clark) Yang
>>>
>>>
>>> 2013/6/30 Pramod N <np...@gmail.com>
>>>
>>>> Hi Sam,
>>>> This might interest you.
>>>> https://github.com/clarkyzl/mpich2-yarn
>>>>
>>>>
>>>>
>>>> Pramod N <http://atmachinelearner.blogspot.in>
>>>> Bruce Wayne of web
>>>> @machinelearner <https://twitter.com/machinelearner>
>>>>
>>>> --
>>>>
>>>>
>>>> On Sun, Jun 30, 2013 at 6:02 PM, sam liu <sa...@gmail.com>wrote:
>>>>
>>>>> Hi Experts,
>>>>>
>>>>> Does Hadoop 2.0.5-alpha supports MPI programs?
>>>>>
>>>>> - If yes, is there any example of writing a MPI program on Yarn? How
>>>>> to write the client side code to configure and submit the MPI job?
>>>>> - If no, which Hadoop version will support MPI programming?
>>>>>
>>>>> Thanks!
>>>>>
>>>>
>>>>
>>>
>>
>
--
Olivier Renault
Solution Engineer - Big Data - Hortonworks, Inc.
+44 7500 933 036
orenault@hortonworks.com
www.hortonworks.com
<http://hortonworks.com/products/hortonworks-sandbox/>
Re: How to write/run MPI program on Yarn?
Posted by sam liu <sa...@gmail.com>.
Any one could help answer above questions? Thanks a lot!
2013/7/1 sam liu <sa...@gmail.com>
> Thanks Pramod and Clark!
>
> 1. What's the relationship of Hadoop 2.x branch and mpich2-yarn project?
> 2. Does Hadoop 2.x branch plan to include MPI implementation? I mentioned
> there is already a JIRA:
> https://issues.apache.org/jira/browse/MAPREDUCE-2911
>
>
> 2013/7/1 Clark Yang (杨卓荦) <ya...@gmail.com>
>
> Hi, sam
>>
>> Please try it following the README.md on
>> https://github.com/clarkyzl/mpich2-yarn/.
>> The design is simple ,make each nodemanger runs a "smpd" daemon and
>> make application master run "mpiexec" to submit mpi jobs. The biggest
>> problem here, I think, is the yarn scheduler do not support gang
>> secheduling algorithm, and the scheduling is very very slow.
>>
>> Any ideas please feel free to share.
>>
>> Cheers,
>> Zhuoluo (Clark) Yang
>>
>>
>> 2013/6/30 Pramod N <np...@gmail.com>
>>
>>> Hi Sam,
>>> This might interest you.
>>> https://github.com/clarkyzl/mpich2-yarn
>>>
>>>
>>>
>>> Pramod N <http://atmachinelearner.blogspot.in>
>>> Bruce Wayne of web
>>> @machinelearner <https://twitter.com/machinelearner>
>>>
>>> --
>>>
>>>
>>> On Sun, Jun 30, 2013 at 6:02 PM, sam liu <sa...@gmail.com> wrote:
>>>
>>>> Hi Experts,
>>>>
>>>> Does Hadoop 2.0.5-alpha supports MPI programs?
>>>>
>>>> - If yes, is there any example of writing a MPI program on Yarn? How to
>>>> write the client side code to configure and submit the MPI job?
>>>> - If no, which Hadoop version will support MPI programming?
>>>>
>>>> Thanks!
>>>>
>>>
>>>
>>
>
Re: How to write/run MPI program on Yarn?
Posted by sam liu <sa...@gmail.com>.
Any one could help answer above questions? Thanks a lot!
2013/7/1 sam liu <sa...@gmail.com>
> Thanks Pramod and Clark!
>
> 1. What's the relationship of Hadoop 2.x branch and mpich2-yarn project?
> 2. Does Hadoop 2.x branch plan to include MPI implementation? I mentioned
> there is already a JIRA:
> https://issues.apache.org/jira/browse/MAPREDUCE-2911
>
>
> 2013/7/1 Clark Yang (杨卓荦) <ya...@gmail.com>
>
> Hi, sam
>>
>> Please try it following the README.md on
>> https://github.com/clarkyzl/mpich2-yarn/.
>> The design is simple ,make each nodemanger runs a "smpd" daemon and
>> make application master run "mpiexec" to submit mpi jobs. The biggest
>> problem here, I think, is the yarn scheduler do not support gang
>> secheduling algorithm, and the scheduling is very very slow.
>>
>> Any ideas please feel free to share.
>>
>> Cheers,
>> Zhuoluo (Clark) Yang
>>
>>
>> 2013/6/30 Pramod N <np...@gmail.com>
>>
>>> Hi Sam,
>>> This might interest you.
>>> https://github.com/clarkyzl/mpich2-yarn
>>>
>>>
>>>
>>> Pramod N <http://atmachinelearner.blogspot.in>
>>> Bruce Wayne of web
>>> @machinelearner <https://twitter.com/machinelearner>
>>>
>>> --
>>>
>>>
>>> On Sun, Jun 30, 2013 at 6:02 PM, sam liu <sa...@gmail.com> wrote:
>>>
>>>> Hi Experts,
>>>>
>>>> Does Hadoop 2.0.5-alpha supports MPI programs?
>>>>
>>>> - If yes, is there any example of writing a MPI program on Yarn? How to
>>>> write the client side code to configure and submit the MPI job?
>>>> - If no, which Hadoop version will support MPI programming?
>>>>
>>>> Thanks!
>>>>
>>>
>>>
>>
>
Re: How to write/run MPI program on Yarn?
Posted by sam liu <sa...@gmail.com>.
Any one could help answer above questions? Thanks a lot!
2013/7/1 sam liu <sa...@gmail.com>
> Thanks Pramod and Clark!
>
> 1. What's the relationship of Hadoop 2.x branch and mpich2-yarn project?
> 2. Does Hadoop 2.x branch plan to include MPI implementation? I mentioned
> there is already a JIRA:
> https://issues.apache.org/jira/browse/MAPREDUCE-2911
>
>
> 2013/7/1 Clark Yang (杨卓荦) <ya...@gmail.com>
>
> Hi, sam
>>
>> Please try it following the README.md on
>> https://github.com/clarkyzl/mpich2-yarn/.
>> The design is simple ,make each nodemanger runs a "smpd" daemon and
>> make application master run "mpiexec" to submit mpi jobs. The biggest
>> problem here, I think, is the yarn scheduler do not support gang
>> secheduling algorithm, and the scheduling is very very slow.
>>
>> Any ideas please feel free to share.
>>
>> Cheers,
>> Zhuoluo (Clark) Yang
>>
>>
>> 2013/6/30 Pramod N <np...@gmail.com>
>>
>>> Hi Sam,
>>> This might interest you.
>>> https://github.com/clarkyzl/mpich2-yarn
>>>
>>>
>>>
>>> Pramod N <http://atmachinelearner.blogspot.in>
>>> Bruce Wayne of web
>>> @machinelearner <https://twitter.com/machinelearner>
>>>
>>> --
>>>
>>>
>>> On Sun, Jun 30, 2013 at 6:02 PM, sam liu <sa...@gmail.com> wrote:
>>>
>>>> Hi Experts,
>>>>
>>>> Does Hadoop 2.0.5-alpha supports MPI programs?
>>>>
>>>> - If yes, is there any example of writing a MPI program on Yarn? How to
>>>> write the client side code to configure and submit the MPI job?
>>>> - If no, which Hadoop version will support MPI programming?
>>>>
>>>> Thanks!
>>>>
>>>
>>>
>>
>
Re: How to write/run MPI program on Yarn?
Posted by sam liu <sa...@gmail.com>.
Any one could help answer above questions? Thanks a lot!
2013/7/1 sam liu <sa...@gmail.com>
> Thanks Pramod and Clark!
>
> 1. What's the relationship of Hadoop 2.x branch and mpich2-yarn project?
> 2. Does Hadoop 2.x branch plan to include MPI implementation? I mentioned
> there is already a JIRA:
> https://issues.apache.org/jira/browse/MAPREDUCE-2911
>
>
> 2013/7/1 Clark Yang (杨卓荦) <ya...@gmail.com>
>
> Hi, sam
>>
>> Please try it following the README.md on
>> https://github.com/clarkyzl/mpich2-yarn/.
>> The design is simple ,make each nodemanger runs a "smpd" daemon and
>> make application master run "mpiexec" to submit mpi jobs. The biggest
>> problem here, I think, is the yarn scheduler do not support gang
>> secheduling algorithm, and the scheduling is very very slow.
>>
>> Any ideas please feel free to share.
>>
>> Cheers,
>> Zhuoluo (Clark) Yang
>>
>>
>> 2013/6/30 Pramod N <np...@gmail.com>
>>
>>> Hi Sam,
>>> This might interest you.
>>> https://github.com/clarkyzl/mpich2-yarn
>>>
>>>
>>>
>>> Pramod N <http://atmachinelearner.blogspot.in>
>>> Bruce Wayne of web
>>> @machinelearner <https://twitter.com/machinelearner>
>>>
>>> --
>>>
>>>
>>> On Sun, Jun 30, 2013 at 6:02 PM, sam liu <sa...@gmail.com> wrote:
>>>
>>>> Hi Experts,
>>>>
>>>> Does Hadoop 2.0.5-alpha supports MPI programs?
>>>>
>>>> - If yes, is there any example of writing a MPI program on Yarn? How to
>>>> write the client side code to configure and submit the MPI job?
>>>> - If no, which Hadoop version will support MPI programming?
>>>>
>>>> Thanks!
>>>>
>>>
>>>
>>
>
Re: How to write/run MPI program on Yarn?
Posted by sam liu <sa...@gmail.com>.
Thanks Pramod and Clark!
1. What's the relationship of Hadoop 2.x branch and mpich2-yarn project?
2. Does Hadoop 2.x branch plan to include MPI implementation? I mentioned
there is already a JIRA:
https://issues.apache.org/jira/browse/MAPREDUCE-2911
2013/7/1 Clark Yang (杨卓荦) <ya...@gmail.com>
> Hi, sam
>
> Please try it following the README.md on
> https://github.com/clarkyzl/mpich2-yarn/.
> The design is simple ,make each nodemanger runs a "smpd" daemon and
> make application master run "mpiexec" to submit mpi jobs. The biggest
> problem here, I think, is the yarn scheduler do not support gang
> secheduling algorithm, and the scheduling is very very slow.
>
> Any ideas please feel free to share.
>
> Cheers,
> Zhuoluo (Clark) Yang
>
>
> 2013/6/30 Pramod N <np...@gmail.com>
>
>> Hi Sam,
>> This might interest you.
>> https://github.com/clarkyzl/mpich2-yarn
>>
>>
>>
>> Pramod N <http://atmachinelearner.blogspot.in>
>> Bruce Wayne of web
>> @machinelearner <https://twitter.com/machinelearner>
>>
>> --
>>
>>
>> On Sun, Jun 30, 2013 at 6:02 PM, sam liu <sa...@gmail.com> wrote:
>>
>>> Hi Experts,
>>>
>>> Does Hadoop 2.0.5-alpha supports MPI programs?
>>>
>>> - If yes, is there any example of writing a MPI program on Yarn? How to
>>> write the client side code to configure and submit the MPI job?
>>> - If no, which Hadoop version will support MPI programming?
>>>
>>> Thanks!
>>>
>>
>>
>
New LinkedIn Group: Cloud & Big Data Consumers and Service Providers
Posted by naser chowdhury <as...@yahoo.com>.
Hi All,
Just opened up a new LinkedIn group: Cloud & Big Data Consumers and Service Providers. Please feel free to join to discuss issues related to those topics.
http://www.linkedin.com/groups?home=&gid=5084326&trk=anet_ug_hm
Thanks,
Naser Chowdhury,
Ph.D. Candidate in Information Technology
Re: How to write/run MPI program on Yarn?
Posted by sam liu <sa...@gmail.com>.
Thanks Pramod and Clark!
1. What's the relationship of Hadoop 2.x branch and mpich2-yarn project?
2. Does Hadoop 2.x branch plan to include MPI implementation? I mentioned
there is already a JIRA:
https://issues.apache.org/jira/browse/MAPREDUCE-2911
2013/7/1 Clark Yang (杨卓荦) <ya...@gmail.com>
> Hi, sam
>
> Please try it following the README.md on
> https://github.com/clarkyzl/mpich2-yarn/.
> The design is simple ,make each nodemanger runs a "smpd" daemon and
> make application master run "mpiexec" to submit mpi jobs. The biggest
> problem here, I think, is the yarn scheduler do not support gang
> secheduling algorithm, and the scheduling is very very slow.
>
> Any ideas please feel free to share.
>
> Cheers,
> Zhuoluo (Clark) Yang
>
>
> 2013/6/30 Pramod N <np...@gmail.com>
>
>> Hi Sam,
>> This might interest you.
>> https://github.com/clarkyzl/mpich2-yarn
>>
>>
>>
>> Pramod N <http://atmachinelearner.blogspot.in>
>> Bruce Wayne of web
>> @machinelearner <https://twitter.com/machinelearner>
>>
>> --
>>
>>
>> On Sun, Jun 30, 2013 at 6:02 PM, sam liu <sa...@gmail.com> wrote:
>>
>>> Hi Experts,
>>>
>>> Does Hadoop 2.0.5-alpha supports MPI programs?
>>>
>>> - If yes, is there any example of writing a MPI program on Yarn? How to
>>> write the client side code to configure and submit the MPI job?
>>> - If no, which Hadoop version will support MPI programming?
>>>
>>> Thanks!
>>>
>>
>>
>
Re: How to write/run MPI program on Yarn?
Posted by sam liu <sa...@gmail.com>.
Thanks Pramod and Clark!
1. What's the relationship of Hadoop 2.x branch and mpich2-yarn project?
2. Does Hadoop 2.x branch plan to include MPI implementation? I mentioned
there is already a JIRA:
https://issues.apache.org/jira/browse/MAPREDUCE-2911
2013/7/1 Clark Yang (杨卓荦) <ya...@gmail.com>
> Hi, sam
>
> Please try it following the README.md on
> https://github.com/clarkyzl/mpich2-yarn/.
> The design is simple ,make each nodemanger runs a "smpd" daemon and
> make application master run "mpiexec" to submit mpi jobs. The biggest
> problem here, I think, is the yarn scheduler do not support gang
> secheduling algorithm, and the scheduling is very very slow.
>
> Any ideas please feel free to share.
>
> Cheers,
> Zhuoluo (Clark) Yang
>
>
> 2013/6/30 Pramod N <np...@gmail.com>
>
>> Hi Sam,
>> This might interest you.
>> https://github.com/clarkyzl/mpich2-yarn
>>
>>
>>
>> Pramod N <http://atmachinelearner.blogspot.in>
>> Bruce Wayne of web
>> @machinelearner <https://twitter.com/machinelearner>
>>
>> --
>>
>>
>> On Sun, Jun 30, 2013 at 6:02 PM, sam liu <sa...@gmail.com> wrote:
>>
>>> Hi Experts,
>>>
>>> Does Hadoop 2.0.5-alpha supports MPI programs?
>>>
>>> - If yes, is there any example of writing a MPI program on Yarn? How to
>>> write the client side code to configure and submit the MPI job?
>>> - If no, which Hadoop version will support MPI programming?
>>>
>>> Thanks!
>>>
>>
>>
>
Re: How to write/run MPI program on Yarn?
Posted by sam liu <sa...@gmail.com>.
Thanks Pramod and Clark!
1. What's the relationship of Hadoop 2.x branch and mpich2-yarn project?
2. Does Hadoop 2.x branch plan to include MPI implementation? I mentioned
there is already a JIRA:
https://issues.apache.org/jira/browse/MAPREDUCE-2911
2013/7/1 Clark Yang (杨卓荦) <ya...@gmail.com>
> Hi, sam
>
> Please try it following the README.md on
> https://github.com/clarkyzl/mpich2-yarn/.
> The design is simple ,make each nodemanger runs a "smpd" daemon and
> make application master run "mpiexec" to submit mpi jobs. The biggest
> problem here, I think, is the yarn scheduler do not support gang
> secheduling algorithm, and the scheduling is very very slow.
>
> Any ideas please feel free to share.
>
> Cheers,
> Zhuoluo (Clark) Yang
>
>
> 2013/6/30 Pramod N <np...@gmail.com>
>
>> Hi Sam,
>> This might interest you.
>> https://github.com/clarkyzl/mpich2-yarn
>>
>>
>>
>> Pramod N <http://atmachinelearner.blogspot.in>
>> Bruce Wayne of web
>> @machinelearner <https://twitter.com/machinelearner>
>>
>> --
>>
>>
>> On Sun, Jun 30, 2013 at 6:02 PM, sam liu <sa...@gmail.com> wrote:
>>
>>> Hi Experts,
>>>
>>> Does Hadoop 2.0.5-alpha supports MPI programs?
>>>
>>> - If yes, is there any example of writing a MPI program on Yarn? How to
>>> write the client side code to configure and submit the MPI job?
>>> - If no, which Hadoop version will support MPI programming?
>>>
>>> Thanks!
>>>
>>
>>
>
New LinkedIn Group: Cloud & Big Data Consumers and Service Providers
Posted by naser chowdhury <as...@yahoo.com>.
Hi All,
Just opened up a new LinkedIn group: Cloud & Big Data Consumers and Service Providers. Please feel free to join to discuss issues related to those topics.
http://www.linkedin.com/groups?home=&gid=5084326&trk=anet_ug_hm
Thanks,
Naser Chowdhury,
Ph.D. Candidate in Information Technology
New LinkedIn Group: Cloud & Big Data Consumers and Service Providers
Posted by naser chowdhury <as...@yahoo.com>.
Hi All,
Just opened up a new LinkedIn group: Cloud & Big Data Consumers and Service Providers. Please feel free to join to discuss issues related to those topics.
http://www.linkedin.com/groups?home=&gid=5084326&trk=anet_ug_hm
Thanks,
Naser Chowdhury,
Ph.D. Candidate in Information Technology
New LinkedIn Group: Cloud & Big Data Consumers and Service Providers
Posted by naser chowdhury <as...@yahoo.com>.
Hi All,
Just opened up a new LinkedIn group: Cloud & Big Data Consumers and Service Providers. Please feel free to join to discuss issues related to those topics.
http://www.linkedin.com/groups?home=&gid=5084326&trk=anet_ug_hm
Thanks,
Naser Chowdhury,
Ph.D. Candidate in Information Technology
Re: How to write/run MPI program on Yarn?
Posted by "Clark Yang (杨卓荦)" <ya...@gmail.com>.
Hi, sam
Please try it following the README.md on
https://github.com/clarkyzl/mpich2-yarn/.
The design is simple ,make each nodemanger runs a "smpd" daemon and make
application master run "mpiexec" to submit mpi jobs. The biggest problem
here, I think, is the yarn scheduler do not support gang secheduling
algorithm, and the scheduling is very very slow.
Any ideas please feel free to share.
Cheers,
Zhuoluo (Clark) Yang
2013/6/30 Pramod N <np...@gmail.com>
> Hi Sam,
> This might interest you.
> https://github.com/clarkyzl/mpich2-yarn
>
>
>
> Pramod N <http://atmachinelearner.blogspot.in>
> Bruce Wayne of web
> @machinelearner <https://twitter.com/machinelearner>
>
> --
>
>
> On Sun, Jun 30, 2013 at 6:02 PM, sam liu <sa...@gmail.com> wrote:
>
>> Hi Experts,
>>
>> Does Hadoop 2.0.5-alpha supports MPI programs?
>>
>> - If yes, is there any example of writing a MPI program on Yarn? How to
>> write the client side code to configure and submit the MPI job?
>> - If no, which Hadoop version will support MPI programming?
>>
>> Thanks!
>>
>
>
Re: How to write/run MPI program on Yarn?
Posted by "Clark Yang (杨卓荦)" <ya...@gmail.com>.
Hi, sam
Please try it following the README.md on
https://github.com/clarkyzl/mpich2-yarn/.
The design is simple ,make each nodemanger runs a "smpd" daemon and make
application master run "mpiexec" to submit mpi jobs. The biggest problem
here, I think, is the yarn scheduler do not support gang secheduling
algorithm, and the scheduling is very very slow.
Any ideas please feel free to share.
Cheers,
Zhuoluo (Clark) Yang
2013/6/30 Pramod N <np...@gmail.com>
> Hi Sam,
> This might interest you.
> https://github.com/clarkyzl/mpich2-yarn
>
>
>
> Pramod N <http://atmachinelearner.blogspot.in>
> Bruce Wayne of web
> @machinelearner <https://twitter.com/machinelearner>
>
> --
>
>
> On Sun, Jun 30, 2013 at 6:02 PM, sam liu <sa...@gmail.com> wrote:
>
>> Hi Experts,
>>
>> Does Hadoop 2.0.5-alpha supports MPI programs?
>>
>> - If yes, is there any example of writing a MPI program on Yarn? How to
>> write the client side code to configure and submit the MPI job?
>> - If no, which Hadoop version will support MPI programming?
>>
>> Thanks!
>>
>
>
Re: How to write/run MPI program on Yarn?
Posted by "Clark Yang (杨卓荦)" <ya...@gmail.com>.
Hi, sam
Please try it following the README.md on
https://github.com/clarkyzl/mpich2-yarn/.
The design is simple ,make each nodemanger runs a "smpd" daemon and make
application master run "mpiexec" to submit mpi jobs. The biggest problem
here, I think, is the yarn scheduler do not support gang secheduling
algorithm, and the scheduling is very very slow.
Any ideas please feel free to share.
Cheers,
Zhuoluo (Clark) Yang
2013/6/30 Pramod N <np...@gmail.com>
> Hi Sam,
> This might interest you.
> https://github.com/clarkyzl/mpich2-yarn
>
>
>
> Pramod N <http://atmachinelearner.blogspot.in>
> Bruce Wayne of web
> @machinelearner <https://twitter.com/machinelearner>
>
> --
>
>
> On Sun, Jun 30, 2013 at 6:02 PM, sam liu <sa...@gmail.com> wrote:
>
>> Hi Experts,
>>
>> Does Hadoop 2.0.5-alpha supports MPI programs?
>>
>> - If yes, is there any example of writing a MPI program on Yarn? How to
>> write the client side code to configure and submit the MPI job?
>> - If no, which Hadoop version will support MPI programming?
>>
>> Thanks!
>>
>
>
Re: How to write/run MPI program on Yarn?
Posted by "Clark Yang (杨卓荦)" <ya...@gmail.com>.
Hi, sam
Please try it following the README.md on
https://github.com/clarkyzl/mpich2-yarn/.
The design is simple ,make each nodemanger runs a "smpd" daemon and make
application master run "mpiexec" to submit mpi jobs. The biggest problem
here, I think, is the yarn scheduler do not support gang secheduling
algorithm, and the scheduling is very very slow.
Any ideas please feel free to share.
Cheers,
Zhuoluo (Clark) Yang
2013/6/30 Pramod N <np...@gmail.com>
> Hi Sam,
> This might interest you.
> https://github.com/clarkyzl/mpich2-yarn
>
>
>
> Pramod N <http://atmachinelearner.blogspot.in>
> Bruce Wayne of web
> @machinelearner <https://twitter.com/machinelearner>
>
> --
>
>
> On Sun, Jun 30, 2013 at 6:02 PM, sam liu <sa...@gmail.com> wrote:
>
>> Hi Experts,
>>
>> Does Hadoop 2.0.5-alpha supports MPI programs?
>>
>> - If yes, is there any example of writing a MPI program on Yarn? How to
>> write the client side code to configure and submit the MPI job?
>> - If no, which Hadoop version will support MPI programming?
>>
>> Thanks!
>>
>
>
Re: How to write/run MPI program on Yarn?
Posted by Pramod N <np...@gmail.com>.
Hi Sam,
This might interest you.
https://github.com/clarkyzl/mpich2-yarn
Pramod N <http://atmachinelearner.blogspot.in>
Bruce Wayne of web
@machinelearner <https://twitter.com/machinelearner>
--
On Sun, Jun 30, 2013 at 6:02 PM, sam liu <sa...@gmail.com> wrote:
> Hi Experts,
>
> Does Hadoop 2.0.5-alpha supports MPI programs?
>
> - If yes, is there any example of writing a MPI program on Yarn? How to
> write the client side code to configure and submit the MPI job?
> - If no, which Hadoop version will support MPI programming?
>
> Thanks!
>
Re: How to write/run MPI program on Yarn?
Posted by Pramod N <np...@gmail.com>.
Hi Sam,
This might interest you.
https://github.com/clarkyzl/mpich2-yarn
Pramod N <http://atmachinelearner.blogspot.in>
Bruce Wayne of web
@machinelearner <https://twitter.com/machinelearner>
--
On Sun, Jun 30, 2013 at 6:02 PM, sam liu <sa...@gmail.com> wrote:
> Hi Experts,
>
> Does Hadoop 2.0.5-alpha supports MPI programs?
>
> - If yes, is there any example of writing a MPI program on Yarn? How to
> write the client side code to configure and submit the MPI job?
> - If no, which Hadoop version will support MPI programming?
>
> Thanks!
>
Re: How to write/run MPI program on Yarn?
Posted by Pramod N <np...@gmail.com>.
Hi Sam,
This might interest you.
https://github.com/clarkyzl/mpich2-yarn
Pramod N <http://atmachinelearner.blogspot.in>
Bruce Wayne of web
@machinelearner <https://twitter.com/machinelearner>
--
On Sun, Jun 30, 2013 at 6:02 PM, sam liu <sa...@gmail.com> wrote:
> Hi Experts,
>
> Does Hadoop 2.0.5-alpha supports MPI programs?
>
> - If yes, is there any example of writing a MPI program on Yarn? How to
> write the client side code to configure and submit the MPI job?
> - If no, which Hadoop version will support MPI programming?
>
> Thanks!
>
Re: How to write/run MPI program on Yarn?
Posted by Pramod N <np...@gmail.com>.
Hi Sam,
This might interest you.
https://github.com/clarkyzl/mpich2-yarn
Pramod N <http://atmachinelearner.blogspot.in>
Bruce Wayne of web
@machinelearner <https://twitter.com/machinelearner>
--
On Sun, Jun 30, 2013 at 6:02 PM, sam liu <sa...@gmail.com> wrote:
> Hi Experts,
>
> Does Hadoop 2.0.5-alpha supports MPI programs?
>
> - If yes, is there any example of writing a MPI program on Yarn? How to
> write the client side code to configure and submit the MPI job?
> - If no, which Hadoop version will support MPI programming?
>
> Thanks!
>