You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-dev@hadoop.apache.org by "Milind Bhandarkar (JIRA)" <ji...@apache.org> on 2011/08/30 09:04:38 UTC
[jira] [Created] (MAPREDUCE-2911) Hamster: Hadoop And Mpi on the
same cluSTER
Hamster: Hadoop And Mpi on the same cluSTER
-------------------------------------------
Key: MAPREDUCE-2911
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2911
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: mrv2
Affects Versions: 0.23.0
Environment: All Unix-Environments
Reporter: Milind Bhandarkar
Assignee: Milind Bhandarkar
Fix For: 0.23.0
MPI is commonly used for many machine-learning applications. OpenMPI (http://www.open-mpi.org/) is a popular BSD-licensed version of MPI. In the past, running MPI application on a Hadoop cluster was achieved using Hadoop Streaming (http://videolectures.net/nipsworkshops2010_ye_gbd/), but it was kludgy. After the resource-manager separation from JobTracker in Hadoop, we have all the tools needed to make MPI a first-class citizen on a Hadoop cluster. I am currently working on the patch to make MPI an application-master. Initial version of this patch will be available soon (hopefully before September 10.) This jira will track the development of Hamster: The application master for MPI.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] [Created] (MAPREDUCE-2911) Hamster: Hadoop And Mpi on the
same cluSTER
Posted by Josh Patterson <jo...@cloudera.com>.
I've also heard that Matei Z is working on moving Spark to MRv2, but I
havent confirmed that yet.
JP
On Thu, Sep 1, 2011 at 12:22 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
> Those are the two public ones I know of currently...
>
> On Aug 31, 2011, at 7:22 PM, Josh Patterson wrote:
>
>> Do we have a list of all MR2 frameworks being worked on currently
>> beyond MPI and Spark?
>>
>> JP
>>
>> On Tue, Aug 30, 2011 at 3:04 AM, Milind Bhandarkar (JIRA)
>> <ji...@apache.org> wrote:
>>> Hamster: Hadoop And Mpi on the same cluSTER
>>> -------------------------------------------
>>>
>>> Key: MAPREDUCE-2911
>>> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2911
>>> Project: Hadoop Map/Reduce
>>> Issue Type: New Feature
>>> Components: mrv2
>>> Affects Versions: 0.23.0
>>> Environment: All Unix-Environments
>>> Reporter: Milind Bhandarkar
>>> Assignee: Milind Bhandarkar
>>> Fix For: 0.23.0
>>>
>>>
>>> MPI is commonly used for many machine-learning applications. OpenMPI (http://www.open-mpi.org/) is a popular BSD-licensed version of MPI. In the past, running MPI application on a Hadoop cluster was achieved using Hadoop Streaming (http://videolectures.net/nipsworkshops2010_ye_gbd/), but it was kludgy. After the resource-manager separation from JobTracker in Hadoop, we have all the tools needed to make MPI a first-class citizen on a Hadoop cluster. I am currently working on the patch to make MPI an application-master. Initial version of this patch will be available soon (hopefully before September 10.) This jira will track the development of Hamster: The application master for MPI.
>>>
>>> --
>>> This message is automatically generated by JIRA.
>>> For more information on JIRA, see: http://www.atlassian.com/software/jira
>>>
>>>
>>>
>>
>>
>>
>> --
>> Twitter: @jpatanooga
>> Solution Architect @ Cloudera
>> hadoop: http://www.cloudera.com
>
>
--
Twitter: @jpatanooga
Solution Architect @ Cloudera
hadoop: http://www.cloudera.com
Re: [jira] [Created] (MAPREDUCE-2911) Hamster: Hadoop And Mpi on the
same cluSTER
Posted by "Edward J. Yoon" <ed...@apache.org>.
https://issues.apache.org/jira/browse/HAMA-431
On Thu, Sep 1, 2011 at 3:13 PM, Todd Lipcon <to...@cloudera.com> wrote:
> Is Golden Orb MR2ifying? http://www.goldenorbos.org/
>
> On Wed, Aug 31, 2011 at 9:22 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
>> Those are the two public ones I know of currently...
>>
>> On Aug 31, 2011, at 7:22 PM, Josh Patterson wrote:
>>
>>> Do we have a list of all MR2 frameworks being worked on currently
>>> beyond MPI and Spark?
>>>
>>> JP
>>>
>>> On Tue, Aug 30, 2011 at 3:04 AM, Milind Bhandarkar (JIRA)
>>> <ji...@apache.org> wrote:
>>>> Hamster: Hadoop And Mpi on the same cluSTER
>>>> -------------------------------------------
>>>>
>>>> Key: MAPREDUCE-2911
>>>> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2911
>>>> Project: Hadoop Map/Reduce
>>>> Issue Type: New Feature
>>>> Components: mrv2
>>>> Affects Versions: 0.23.0
>>>> Environment: All Unix-Environments
>>>> Reporter: Milind Bhandarkar
>>>> Assignee: Milind Bhandarkar
>>>> Fix For: 0.23.0
>>>>
>>>>
>>>> MPI is commonly used for many machine-learning applications. OpenMPI (http://www.open-mpi.org/) is a popular BSD-licensed version of MPI. In the past, running MPI application on a Hadoop cluster was achieved using Hadoop Streaming (http://videolectures.net/nipsworkshops2010_ye_gbd/), but it was kludgy. After the resource-manager separation from JobTracker in Hadoop, we have all the tools needed to make MPI a first-class citizen on a Hadoop cluster. I am currently working on the patch to make MPI an application-master. Initial version of this patch will be available soon (hopefully before September 10.) This jira will track the development of Hamster: The application master for MPI.
>>>>
>>>> --
>>>> This message is automatically generated by JIRA.
>>>> For more information on JIRA, see: http://www.atlassian.com/software/jira
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Twitter: @jpatanooga
>>> Solution Architect @ Cloudera
>>> hadoop: http://www.cloudera.com
>>
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
--
Best Regards, Edward J. Yoon
@eddieyoon
Re: [jira] [Created] (MAPREDUCE-2911) Hamster: Hadoop And Mpi on the
same cluSTER
Posted by Todd Lipcon <to...@cloudera.com>.
Is Golden Orb MR2ifying? http://www.goldenorbos.org/
On Wed, Aug 31, 2011 at 9:22 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
> Those are the two public ones I know of currently...
>
> On Aug 31, 2011, at 7:22 PM, Josh Patterson wrote:
>
>> Do we have a list of all MR2 frameworks being worked on currently
>> beyond MPI and Spark?
>>
>> JP
>>
>> On Tue, Aug 30, 2011 at 3:04 AM, Milind Bhandarkar (JIRA)
>> <ji...@apache.org> wrote:
>>> Hamster: Hadoop And Mpi on the same cluSTER
>>> -------------------------------------------
>>>
>>> Key: MAPREDUCE-2911
>>> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2911
>>> Project: Hadoop Map/Reduce
>>> Issue Type: New Feature
>>> Components: mrv2
>>> Affects Versions: 0.23.0
>>> Environment: All Unix-Environments
>>> Reporter: Milind Bhandarkar
>>> Assignee: Milind Bhandarkar
>>> Fix For: 0.23.0
>>>
>>>
>>> MPI is commonly used for many machine-learning applications. OpenMPI (http://www.open-mpi.org/) is a popular BSD-licensed version of MPI. In the past, running MPI application on a Hadoop cluster was achieved using Hadoop Streaming (http://videolectures.net/nipsworkshops2010_ye_gbd/), but it was kludgy. After the resource-manager separation from JobTracker in Hadoop, we have all the tools needed to make MPI a first-class citizen on a Hadoop cluster. I am currently working on the patch to make MPI an application-master. Initial version of this patch will be available soon (hopefully before September 10.) This jira will track the development of Hamster: The application master for MPI.
>>>
>>> --
>>> This message is automatically generated by JIRA.
>>> For more information on JIRA, see: http://www.atlassian.com/software/jira
>>>
>>>
>>>
>>
>>
>>
>> --
>> Twitter: @jpatanooga
>> Solution Architect @ Cloudera
>> hadoop: http://www.cloudera.com
>
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: [jira] [Created] (MAPREDUCE-2911) Hamster: Hadoop And Mpi on the same cluSTER
Posted by Arun C Murthy <ac...@hortonworks.com>.
Those are the two public ones I know of currently...
On Aug 31, 2011, at 7:22 PM, Josh Patterson wrote:
> Do we have a list of all MR2 frameworks being worked on currently
> beyond MPI and Spark?
>
> JP
>
> On Tue, Aug 30, 2011 at 3:04 AM, Milind Bhandarkar (JIRA)
> <ji...@apache.org> wrote:
>> Hamster: Hadoop And Mpi on the same cluSTER
>> -------------------------------------------
>>
>> Key: MAPREDUCE-2911
>> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2911
>> Project: Hadoop Map/Reduce
>> Issue Type: New Feature
>> Components: mrv2
>> Affects Versions: 0.23.0
>> Environment: All Unix-Environments
>> Reporter: Milind Bhandarkar
>> Assignee: Milind Bhandarkar
>> Fix For: 0.23.0
>>
>>
>> MPI is commonly used for many machine-learning applications. OpenMPI (http://www.open-mpi.org/) is a popular BSD-licensed version of MPI. In the past, running MPI application on a Hadoop cluster was achieved using Hadoop Streaming (http://videolectures.net/nipsworkshops2010_ye_gbd/), but it was kludgy. After the resource-manager separation from JobTracker in Hadoop, we have all the tools needed to make MPI a first-class citizen on a Hadoop cluster. I am currently working on the patch to make MPI an application-master. Initial version of this patch will be available soon (hopefully before September 10.) This jira will track the development of Hamster: The application master for MPI.
>>
>> --
>> This message is automatically generated by JIRA.
>> For more information on JIRA, see: http://www.atlassian.com/software/jira
>>
>>
>>
>
>
>
> --
> Twitter: @jpatanooga
> Solution Architect @ Cloudera
> hadoop: http://www.cloudera.com
Re: [jira] [Created] (MAPREDUCE-2911) Hamster: Hadoop And Mpi on the
same cluSTER
Posted by Owen O'Malley <ow...@hortonworks.com>.
On Wed, Aug 31, 2011 at 7:22 PM, Josh Patterson <jo...@cloudera.com> wrote:
> Do we have a list of all MR2 frameworks being worked on currently
> beyond MPI and Spark?
Giraph is also going to port over:
https://issues.apache.org/jira/browse/GIRAPH-13
-- Owen
Re: [jira] [Created] (MAPREDUCE-2911) Hamster: Hadoop And Mpi on the same cluSTER
Posted by Arun C Murthy <ac...@hortonworks.com>.
I've started a wiki to track this, please help keep it up-to-date...
http://wiki.apache.org/hadoop/PoweredByYarn
thanks,
Arun
On Aug 31, 2011, at 7:22 PM, Josh Patterson wrote:
> Do we have a list of all MR2 frameworks being worked on currently
> beyond MPI and Spark?
>
> JP
>
> On Tue, Aug 30, 2011 at 3:04 AM, Milind Bhandarkar (JIRA)
> <ji...@apache.org> wrote:
>> Hamster: Hadoop And Mpi on the same cluSTER
>> -------------------------------------------
>>
>> Key: MAPREDUCE-2911
>> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2911
>> Project: Hadoop Map/Reduce
>> Issue Type: New Feature
>> Components: mrv2
>> Affects Versions: 0.23.0
>> Environment: All Unix-Environments
>> Reporter: Milind Bhandarkar
>> Assignee: Milind Bhandarkar
>> Fix For: 0.23.0
>>
>>
>> MPI is commonly used for many machine-learning applications. OpenMPI (http://www.open-mpi.org/) is a popular BSD-licensed version of MPI. In the past, running MPI application on a Hadoop cluster was achieved using Hadoop Streaming (http://videolectures.net/nipsworkshops2010_ye_gbd/), but it was kludgy. After the resource-manager separation from JobTracker in Hadoop, we have all the tools needed to make MPI a first-class citizen on a Hadoop cluster. I am currently working on the patch to make MPI an application-master. Initial version of this patch will be available soon (hopefully before September 10.) This jira will track the development of Hamster: The application master for MPI.
>>
>> --
>> This message is automatically generated by JIRA.
>> For more information on JIRA, see: http://www.atlassian.com/software/jira
>>
>>
>>
>
>
>
> --
> Twitter: @jpatanooga
> Solution Architect @ Cloudera
> hadoop: http://www.cloudera.com
Re: [jira] [Created] (MAPREDUCE-2911) Hamster: Hadoop And Mpi on the
same cluSTER
Posted by Josh Patterson <jo...@cloudera.com>.
Do we have a list of all MR2 frameworks being worked on currently
beyond MPI and Spark?
JP
On Tue, Aug 30, 2011 at 3:04 AM, Milind Bhandarkar (JIRA)
<ji...@apache.org> wrote:
> Hamster: Hadoop And Mpi on the same cluSTER
> -------------------------------------------
>
> Key: MAPREDUCE-2911
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2911
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2
> Affects Versions: 0.23.0
> Environment: All Unix-Environments
> Reporter: Milind Bhandarkar
> Assignee: Milind Bhandarkar
> Fix For: 0.23.0
>
>
> MPI is commonly used for many machine-learning applications. OpenMPI (http://www.open-mpi.org/) is a popular BSD-licensed version of MPI. In the past, running MPI application on a Hadoop cluster was achieved using Hadoop Streaming (http://videolectures.net/nipsworkshops2010_ye_gbd/), but it was kludgy. After the resource-manager separation from JobTracker in Hadoop, we have all the tools needed to make MPI a first-class citizen on a Hadoop cluster. I am currently working on the patch to make MPI an application-master. Initial version of this patch will be available soon (hopefully before September 10.) This jira will track the development of Hamster: The application master for MPI.
>
> --
> This message is automatically generated by JIRA.
> For more information on JIRA, see: http://www.atlassian.com/software/jira
>
>
>
--
Twitter: @jpatanooga
Solution Architect @ Cloudera
hadoop: http://www.cloudera.com
[jira] [Reopened] (MAPREDUCE-2911) Hamster: Hadoop And Mpi on the
same cluSTER
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy reopened MAPREDUCE-2911:
--------------------------------------
> Hamster: Hadoop And Mpi on the same cluSTER
> -------------------------------------------
>
> Key: MAPREDUCE-2911
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2911
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2
> Affects Versions: 0.23.0
> Environment: All Unix-Environments
> Reporter: Milind Bhandarkar
> Assignee: Ralph H Castain
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> MPI is commonly used for many machine-learning applications. OpenMPI (http://www.open-mpi.org/) is a popular BSD-licensed version of MPI. In the past, running MPI application on a Hadoop cluster was achieved using Hadoop Streaming (http://videolectures.net/nipsworkshops2010_ye_gbd/), but it was kludgy. After the resource-manager separation from JobTracker in Hadoop, we have all the tools needed to make MPI a first-class citizen on a Hadoop cluster. I am currently working on the patch to make MPI an application-master. Initial version of this patch will be available soon (hopefully before September 10.) This jira will track the development of Hamster: The application master for MPI.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2911) Hamster: Hadoop And Mpi on the
same cluSTER
Posted by "Milind Bhandarkar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Milind Bhandarkar resolved MAPREDUCE-2911.
------------------------------------------
Resolution: Fixed
> Hamster: Hadoop And Mpi on the same cluSTER
> -------------------------------------------
>
> Key: MAPREDUCE-2911
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2911
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2
> Affects Versions: 0.23.0
> Environment: All Unix-Environments
> Reporter: Milind Bhandarkar
> Assignee: Ralph H Castain
> Fix For: 0.24.0
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> MPI is commonly used for many machine-learning applications. OpenMPI (http://www.open-mpi.org/) is a popular BSD-licensed version of MPI. In the past, running MPI application on a Hadoop cluster was achieved using Hadoop Streaming (http://videolectures.net/nipsworkshops2010_ye_gbd/), but it was kludgy. After the resource-manager separation from JobTracker in Hadoop, we have all the tools needed to make MPI a first-class citizen on a Hadoop cluster. I am currently working on the patch to make MPI an application-master. Initial version of this patch will be available soon (hopefully before September 10.) This jira will track the development of Hamster: The application master for MPI.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2911) Hamster: Hadoop And Mpi on the
same cluSTER
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy resolved MAPREDUCE-2911.
--------------------------------------
Resolution: Later
Fix Version/s: (was: 0.24.0)
bq. I was curious to see the code. Which branch has it been committed to?
To clarify, there was no code committed to Hadoop itself.
Ralph - Can you please share information on where it was committed to OpenMPI and how folks can try it out? Thanks.
> Hamster: Hadoop And Mpi on the same cluSTER
> -------------------------------------------
>
> Key: MAPREDUCE-2911
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2911
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2
> Affects Versions: 0.23.0
> Environment: All Unix-Environments
> Reporter: Milind Bhandarkar
> Assignee: Ralph H Castain
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> MPI is commonly used for many machine-learning applications. OpenMPI (http://www.open-mpi.org/) is a popular BSD-licensed version of MPI. In the past, running MPI application on a Hadoop cluster was achieved using Hadoop Streaming (http://videolectures.net/nipsworkshops2010_ye_gbd/), but it was kludgy. After the resource-manager separation from JobTracker in Hadoop, we have all the tools needed to make MPI a first-class citizen on a Hadoop cluster. I am currently working on the patch to make MPI an application-master. Initial version of this patch will be available soon (hopefully before September 10.) This jira will track the development of Hamster: The application master for MPI.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira