You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-dev@hadoop.apache.org by Vinod Kumar Vavilapalli <vi...@hortonworks.com> on 2011/08/18 14:49:35 UTC
Notes for working on mapreduce trunk after the MR-279 merge.
MR-279 branch is merged into mapreduce trunk and this changes things a
bit for developing on mapreduce.
You can get all the help that is needed from the INSTALL file at
http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce/INSTALL.
Reproducing some of those contents here for the short-term lookup.
------------------------------------
Checking out source code
------------------------------------
svn checkout http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce
--------------------------
Directory structure
--------------------------
trunk/
- hadoop-mapreduce ( was mapreduce before)
trunk/hadoop-mapreduce - Classic code. JT/TT reside here
- build.xml
- src
trunk/hadoop-mapreduce/ - New code related to yarn reside here.
- assembly
- pom.xml
- hadoop-mr-client
- hadoop-yarn - Yarn APIs, libraries, and server code
-- hadoop-yarn-api
-- hadoop-yarn-common
-- hadoop-yarn-server - Server code, ResourceManager, NodeManager,
server libraries and tests.
--- hadoop-yarn-server-common
--- hadoop-yarn-server-nodemanager
--- hadoop-yarn-server-resourcemanager
--- hadoop-yarn-server-tests
- hadoop-mr-client - MapReduce server and client code
-- hadoop-mapreduce-client-app
-- hadoop-mapreduce-client-core
-- hadoop-mapreduce-client-jobclient
-- hadoop-mapreduce-client-common
-- hadoop-mapreduce-client-hs
-- hadoop-mapreduce-client-shuffle
-----------
Building
-----------
Building yarn code and install into the local maven cache.
- mvn clean install
- In case you want to skip the tests run: mvn clean install -DskipTests
Building classic code once yarn code is built.
- ant veryclean jar jar-test -Dresolvers=internal
----------
Eclipse
-----------
1) For hacking on the new yarn+MR code in eclipse, you should run
"mvn eclipse:eclipse" and then import the checked out source root as a
maven project.
2) For developing on classic JT/TT code, running "ant eclipse" and
importing as java project should continue to work.
Hope that helps. If you run into issues, please send an email or
create a JIRA issue.
Thanks,
+Vinod
Re: Notes for working on mapreduce trunk after the MR-279 merge.
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
Was off the grid due to illness.
Yes. But we will need separate sections for pre-23 and post-23 Hadoop. Will
do so.
Thanks,
+Vinod
On Thu, Aug 18, 2011 at 11:10 PM, Eli Collins <el...@cloudera.com> wrote:
> On Thu, Aug 18, 2011 at 8:46 AM, Vinod Kumar Vavilapalli
> <vi...@hortonworks.com> wrote:
> > Good idea. Created
> http://wiki.apache.org/hadoop/DevelopingOnTrunkAfter279Merge
> >
>
> Mind updating http://wiki.apache.org/hadoop/HowToContribute with the
> new MR instructions from this page? (would be good to just have one
> set of instructions on how to build hadoop).
>
> Thanks,
> Eli
>
Re: Notes for working on mapreduce trunk after the MR-279 merge.
Posted by Eli Collins <el...@cloudera.com>.
On Thu, Aug 18, 2011 at 8:46 AM, Vinod Kumar Vavilapalli
<vi...@hortonworks.com> wrote:
> Good idea. Created http://wiki.apache.org/hadoop/DevelopingOnTrunkAfter279Merge
>
Mind updating http://wiki.apache.org/hadoop/HowToContribute with the
new MR instructions from this page? (would be good to just have one
set of instructions on how to build hadoop).
Thanks,
Eli
Re: Notes for working on mapreduce trunk after the MR-279 merge.
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
Good idea. Created http://wiki.apache.org/hadoop/DevelopingOnTrunkAfter279Merge
Please feel free to edit it. Once it stabilises, we can move the
content over to HowToContribute and/or something like
HowToBuildAndInstall.
Thanks,
+Vinod
On Thu, Aug 18, 2011 at 7:29 PM, Harsh J <ha...@cloudera.com> wrote:
> Awesome Vinod, thanks much for these instructions!
>
> Perhaps these may be recorded on the frequented wiki HowToContribute
> pages as well? :)
>
> On Thu, Aug 18, 2011 at 6:19 PM, Vinod Kumar Vavilapalli
> <vi...@hortonworks.com> wrote:
>> MR-279 branch is merged into mapreduce trunk and this changes things a
>> bit for developing on mapreduce.
>>
>> You can get all the help that is needed from the INSTALL file at
>> http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce/INSTALL.
>> Reproducing some of those contents here for the short-term lookup.
>>
>> ------------------------------------
>> Checking out source code
>> ------------------------------------
>> svn checkout http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce
>>
>> --------------------------
>> Directory structure
>> --------------------------
>>
>> trunk/
>> - hadoop-mapreduce ( was mapreduce before)
>>
>> trunk/hadoop-mapreduce - Classic code. JT/TT reside here
>> - build.xml
>> - src
>>
>> trunk/hadoop-mapreduce/ - New code related to yarn reside here.
>> - assembly
>> - pom.xml
>> - hadoop-mr-client
>> - hadoop-yarn - Yarn APIs, libraries, and server code
>> -- hadoop-yarn-api
>> -- hadoop-yarn-common
>> -- hadoop-yarn-server - Server code, ResourceManager, NodeManager,
>> server libraries and tests.
>> --- hadoop-yarn-server-common
>> --- hadoop-yarn-server-nodemanager
>> --- hadoop-yarn-server-resourcemanager
>> --- hadoop-yarn-server-tests
>> - hadoop-mr-client - MapReduce server and client code
>> -- hadoop-mapreduce-client-app
>> -- hadoop-mapreduce-client-core
>> -- hadoop-mapreduce-client-jobclient
>> -- hadoop-mapreduce-client-common
>> -- hadoop-mapreduce-client-hs
>> -- hadoop-mapreduce-client-shuffle
>>
>> -----------
>> Building
>> -----------
>> Building yarn code and install into the local maven cache.
>> - mvn clean install
>> - In case you want to skip the tests run: mvn clean install -DskipTests
>>
>> Building classic code once yarn code is built.
>> - ant veryclean jar jar-test -Dresolvers=internal
>>
>> ----------
>> Eclipse
>> -----------
>> 1) For hacking on the new yarn+MR code in eclipse, you should run
>> "mvn eclipse:eclipse" and then import the checked out source root as a
>> maven project.
>> 2) For developing on classic JT/TT code, running "ant eclipse" and
>> importing as java project should continue to work.
>>
>> Hope that helps. If you run into issues, please send an email or
>> create a JIRA issue.
>>
>> Thanks,
>> +Vinod
>>
>
>
>
> --
> Harsh J
>
Re: Notes for working on mapreduce trunk after the MR-279 merge.
Posted by Harsh J <ha...@cloudera.com>.
Awesome Vinod, thanks much for these instructions!
Perhaps these may be recorded on the frequented wiki HowToContribute
pages as well? :)
On Thu, Aug 18, 2011 at 6:19 PM, Vinod Kumar Vavilapalli
<vi...@hortonworks.com> wrote:
> MR-279 branch is merged into mapreduce trunk and this changes things a
> bit for developing on mapreduce.
>
> You can get all the help that is needed from the INSTALL file at
> http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce/INSTALL.
> Reproducing some of those contents here for the short-term lookup.
>
> ------------------------------------
> Checking out source code
> ------------------------------------
> svn checkout http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce
>
> --------------------------
> Directory structure
> --------------------------
>
> trunk/
> - hadoop-mapreduce ( was mapreduce before)
>
> trunk/hadoop-mapreduce - Classic code. JT/TT reside here
> - build.xml
> - src
>
> trunk/hadoop-mapreduce/ - New code related to yarn reside here.
> - assembly
> - pom.xml
> - hadoop-mr-client
> - hadoop-yarn - Yarn APIs, libraries, and server code
> -- hadoop-yarn-api
> -- hadoop-yarn-common
> -- hadoop-yarn-server - Server code, ResourceManager, NodeManager,
> server libraries and tests.
> --- hadoop-yarn-server-common
> --- hadoop-yarn-server-nodemanager
> --- hadoop-yarn-server-resourcemanager
> --- hadoop-yarn-server-tests
> - hadoop-mr-client - MapReduce server and client code
> -- hadoop-mapreduce-client-app
> -- hadoop-mapreduce-client-core
> -- hadoop-mapreduce-client-jobclient
> -- hadoop-mapreduce-client-common
> -- hadoop-mapreduce-client-hs
> -- hadoop-mapreduce-client-shuffle
>
> -----------
> Building
> -----------
> Building yarn code and install into the local maven cache.
> - mvn clean install
> - In case you want to skip the tests run: mvn clean install -DskipTests
>
> Building classic code once yarn code is built.
> - ant veryclean jar jar-test -Dresolvers=internal
>
> ----------
> Eclipse
> -----------
> 1) For hacking on the new yarn+MR code in eclipse, you should run
> "mvn eclipse:eclipse" and then import the checked out source root as a
> maven project.
> 2) For developing on classic JT/TT code, running "ant eclipse" and
> importing as java project should continue to work.
>
> Hope that helps. If you run into issues, please send an email or
> create a JIRA issue.
>
> Thanks,
> +Vinod
>
--
Harsh J
Re: Notes for working on mapreduce trunk after the MR-279 merge.
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
It's working now. I did a pull which shows the following commits:
------------------------------------------------------------------------------------------------------------
commit cec7fc69876fb6bd50612c9282299ac94af6ab70
Author: Vinod Kumar Vavilapalli <vi...@apache.org>
Date: Thu Aug 18 14:10:12 2011 +0000
Updating CHANGES.txt to reflect fixes from MR-279 branch that came
into trunk when merging the two of them.
git-svn-id:
https://svn.apache.org/repos/asf/hadoop/common/trunk@1159232
13f79535-47bb-0310-9956-ffa450edef68
commit 46702790df8d61c18ba8ace768cc7bc6b7318c0a
Author: Vinod Kumar Vavilapalli <vi...@apache.org>
Date: Thu Aug 18 11:07:10 2011 +0000
MAPREDUCE-279. MapReduce 2.0. Merging MR-279 branch into trunk.
Contributed by Arun C Murthy, Christopher Douglas, Devaraj Das, Greg
Roelofs, Jeffrey Naisbitt, Josh Wills, Jo
git-svn-id:
https://svn.apache.org/repos/asf/hadoop/common/trunk@1159166
13f79535-47bb-0310-9956-ffa450edef68
------------------------------------------------------------------------------------------------------------
Thanks,
+Vinod
On Thu, Aug 18, 2011 at 9:24 PM, Vinod Kumar Vavilapalli
<vi...@hortonworks.com> wrote:
> yes.. I also observed this. Looks like the mirrors are setup to use
> the hardcoded path to trunk/mapreduce which changed now to
> trunk/hadoop-mapreduce.
>
> Will see and get the right people to fix it.
>
> Thanks,
> +Vinod
>
> On Thu, Aug 18, 2011 at 9:16 PM, Robert Evans <ev...@yahoo-inc.com> wrote:
>> It looks like git has not seen the changes yet, even though the last change was over 90 mins ago. Is there any way to kick git to pull in the changes sooner so I can rebase?
>>
>> Thanks,
>>
>> Bobby Evans
>>
>> On 8/18/11 7:49 AM, "Vinod Kumar Vavilapalli" <vi...@hortonworks.com> wrote:
>>
>> MR-279 branch is merged into mapreduce trunk and this changes things a
>> bit for developing on mapreduce.
>>
>> You can get all the help that is needed from the INSTALL file at
>> http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce/INSTALL.
>> Reproducing some of those contents here for the short-term lookup.
>>
>> ------------------------------------
>> Checking out source code
>> ------------------------------------
>> svn checkout http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce
>>
>> --------------------------
>> Directory structure
>> --------------------------
>>
>> trunk/
>> - hadoop-mapreduce ( was mapreduce before)
>>
>> trunk/hadoop-mapreduce - Classic code. JT/TT reside here
>> - build.xml
>> - src
>>
>> trunk/hadoop-mapreduce/ - New code related to yarn reside here.
>> - assembly
>> - pom.xml
>> - hadoop-mr-client
>> - hadoop-yarn - Yarn APIs, libraries, and server code
>> -- hadoop-yarn-api
>> -- hadoop-yarn-common
>> -- hadoop-yarn-server - Server code, ResourceManager, NodeManager,
>> server libraries and tests.
>> --- hadoop-yarn-server-common
>> --- hadoop-yarn-server-nodemanager
>> --- hadoop-yarn-server-resourcemanager
>> --- hadoop-yarn-server-tests
>> - hadoop-mr-client - MapReduce server and client code
>> -- hadoop-mapreduce-client-app
>> -- hadoop-mapreduce-client-core
>> -- hadoop-mapreduce-client-jobclient
>> -- hadoop-mapreduce-client-common
>> -- hadoop-mapreduce-client-hs
>> -- hadoop-mapreduce-client-shuffle
>>
>> -----------
>> Building
>> -----------
>> Building yarn code and install into the local maven cache.
>> - mvn clean install
>> - In case you want to skip the tests run: mvn clean install -DskipTests
>>
>> Building classic code once yarn code is built.
>> - ant veryclean jar jar-test -Dresolvers=internal
>>
>> ----------
>> Eclipse
>> -----------
>> 1) For hacking on the new yarn+MR code in eclipse, you should run
>> "mvn eclipse:eclipse" and then import the checked out source root as a
>> maven project.
>> 2) For developing on classic JT/TT code, running "ant eclipse" and
>> importing as java project should continue to work.
>>
>> Hope that helps. If you run into issues, please send an email or
>> create a JIRA issue.
>>
>> Thanks,
>> +Vinod
>>
>>
>
Re: Notes for working on mapreduce trunk after the MR-279 merge.
Posted by Mahadev Konar <ma...@hortonworks.com>.
Thanks Todd.
mahadev
On Aug 19, 2011, at 10:34 AM, Todd Lipcon wrote:
> BTW, I also kicked it manually. I'll try to do it a few times
> throughout the day today until we hear back from Jukka. Feel free to
> ping me if you need me to hit the button.
>
> -Todd
>
> On Fri, Aug 19, 2011 at 10:28 AM, Todd Lipcon <to...@cloudera.com> wrote:
>> Let me ping Jukka and see if he has any idea why it's not auto-updating anymore.
>>
>> -Todd
>>
>> On Fri, Aug 19, 2011 at 10:26 AM, Mahadev Konar <ma...@hortonworks.com> wrote:
>>> I am seeing this issue again. Todd, do we need to fix something else?
>>> I committed 2 patches one yest and one today and I cant get it on the git branch. Something off with git pulling from svn.
>>>
>>> Any other fix we need to do?
>>>
>>>
>>> thanks
>>> mahadev
>>> On Aug 18, 2011, at 10:05 AM, Vinod Kumar Vavilapalli wrote:
>>>
>>>> Thanks Todd!
>>>>
>>>> On Thu, Aug 18, 2011 at 10:27 PM, Todd Lipcon <to...@cloudera.com> wrote:
>>>>> I'm kicking the git mirror now... for some reason, on merges, it often
>>>>> takes several commits before it fully syncs up. I have access to
>>>>> manually poke it into going.
>>>>>
>>>>> -Todd
>>>>>
>>>>> On Thu, Aug 18, 2011 at 8:54 AM, Vinod Kumar Vavilapalli
>>>>> <vi...@hortonworks.com> wrote:
>>>>>> yes.. I also observed this. Looks like the mirrors are setup to use
>>>>>> the hardcoded path to trunk/mapreduce which changed now to
>>>>>> trunk/hadoop-mapreduce.
>>>>>>
>>>>>> Will see and get the right people to fix it.
>>>>>>
>>>>>> Thanks,
>>>>>> +Vinod
>>>>>>
>>>>>> On Thu, Aug 18, 2011 at 9:16 PM, Robert Evans <ev...@yahoo-inc.com> wrote:
>>>>>>> It looks like git has not seen the changes yet, even though the last change was over 90 mins ago. Is there any way to kick git to pull in the changes sooner so I can rebase?
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Bobby Evans
>>>>>>>
>>>>>>> On 8/18/11 7:49 AM, "Vinod Kumar Vavilapalli" <vi...@hortonworks.com> wrote:
>>>>>>>
>>>>>>> MR-279 branch is merged into mapreduce trunk and this changes things a
>>>>>>> bit for developing on mapreduce.
>>>>>>>
>>>>>>> You can get all the help that is needed from the INSTALL file at
>>>>>>> http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce/INSTALL.
>>>>>>> Reproducing some of those contents here for the short-term lookup.
>>>>>>>
>>>>>>> ------------------------------------
>>>>>>> Checking out source code
>>>>>>> ------------------------------------
>>>>>>> svn checkout http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce
>>>>>>>
>>>>>>> --------------------------
>>>>>>> Directory structure
>>>>>>> --------------------------
>>>>>>>
>>>>>>> trunk/
>>>>>>> - hadoop-mapreduce ( was mapreduce before)
>>>>>>>
>>>>>>> trunk/hadoop-mapreduce - Classic code. JT/TT reside here
>>>>>>> - build.xml
>>>>>>> - src
>>>>>>>
>>>>>>> trunk/hadoop-mapreduce/ - New code related to yarn reside here.
>>>>>>> - assembly
>>>>>>> - pom.xml
>>>>>>> - hadoop-mr-client
>>>>>>> - hadoop-yarn - Yarn APIs, libraries, and server code
>>>>>>> -- hadoop-yarn-api
>>>>>>> -- hadoop-yarn-common
>>>>>>> -- hadoop-yarn-server - Server code, ResourceManager, NodeManager,
>>>>>>> server libraries and tests.
>>>>>>> --- hadoop-yarn-server-common
>>>>>>> --- hadoop-yarn-server-nodemanager
>>>>>>> --- hadoop-yarn-server-resourcemanager
>>>>>>> --- hadoop-yarn-server-tests
>>>>>>> - hadoop-mr-client - MapReduce server and client code
>>>>>>> -- hadoop-mapreduce-client-app
>>>>>>> -- hadoop-mapreduce-client-core
>>>>>>> -- hadoop-mapreduce-client-jobclient
>>>>>>> -- hadoop-mapreduce-client-common
>>>>>>> -- hadoop-mapreduce-client-hs
>>>>>>> -- hadoop-mapreduce-client-shuffle
>>>>>>>
>>>>>>> -----------
>>>>>>> Building
>>>>>>> -----------
>>>>>>> Building yarn code and install into the local maven cache.
>>>>>>> - mvn clean install
>>>>>>> - In case you want to skip the tests run: mvn clean install -DskipTests
>>>>>>>
>>>>>>> Building classic code once yarn code is built.
>>>>>>> - ant veryclean jar jar-test -Dresolvers=internal
>>>>>>>
>>>>>>> ----------
>>>>>>> Eclipse
>>>>>>> -----------
>>>>>>> 1) For hacking on the new yarn+MR code in eclipse, you should run
>>>>>>> "mvn eclipse:eclipse" and then import the checked out source root as a
>>>>>>> maven project.
>>>>>>> 2) For developing on classic JT/TT code, running "ant eclipse" and
>>>>>>> importing as java project should continue to work.
>>>>>>>
>>>>>>> Hope that helps. If you run into issues, please send an email or
>>>>>>> create a JIRA issue.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> +Vinod
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Todd Lipcon
>>>>> Software Engineer, Cloudera
>>>>>
>>>
>>>
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
Re: Notes for working on mapreduce trunk after the MR-279 merge.
Posted by Todd Lipcon <to...@cloudera.com>.
BTW, I also kicked it manually. I'll try to do it a few times
throughout the day today until we hear back from Jukka. Feel free to
ping me if you need me to hit the button.
-Todd
On Fri, Aug 19, 2011 at 10:28 AM, Todd Lipcon <to...@cloudera.com> wrote:
> Let me ping Jukka and see if he has any idea why it's not auto-updating anymore.
>
> -Todd
>
> On Fri, Aug 19, 2011 at 10:26 AM, Mahadev Konar <ma...@hortonworks.com> wrote:
>> I am seeing this issue again. Todd, do we need to fix something else?
>> I committed 2 patches one yest and one today and I cant get it on the git branch. Something off with git pulling from svn.
>>
>> Any other fix we need to do?
>>
>>
>> thanks
>> mahadev
>> On Aug 18, 2011, at 10:05 AM, Vinod Kumar Vavilapalli wrote:
>>
>>> Thanks Todd!
>>>
>>> On Thu, Aug 18, 2011 at 10:27 PM, Todd Lipcon <to...@cloudera.com> wrote:
>>>> I'm kicking the git mirror now... for some reason, on merges, it often
>>>> takes several commits before it fully syncs up. I have access to
>>>> manually poke it into going.
>>>>
>>>> -Todd
>>>>
>>>> On Thu, Aug 18, 2011 at 8:54 AM, Vinod Kumar Vavilapalli
>>>> <vi...@hortonworks.com> wrote:
>>>>> yes.. I also observed this. Looks like the mirrors are setup to use
>>>>> the hardcoded path to trunk/mapreduce which changed now to
>>>>> trunk/hadoop-mapreduce.
>>>>>
>>>>> Will see and get the right people to fix it.
>>>>>
>>>>> Thanks,
>>>>> +Vinod
>>>>>
>>>>> On Thu, Aug 18, 2011 at 9:16 PM, Robert Evans <ev...@yahoo-inc.com> wrote:
>>>>>> It looks like git has not seen the changes yet, even though the last change was over 90 mins ago. Is there any way to kick git to pull in the changes sooner so I can rebase?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Bobby Evans
>>>>>>
>>>>>> On 8/18/11 7:49 AM, "Vinod Kumar Vavilapalli" <vi...@hortonworks.com> wrote:
>>>>>>
>>>>>> MR-279 branch is merged into mapreduce trunk and this changes things a
>>>>>> bit for developing on mapreduce.
>>>>>>
>>>>>> You can get all the help that is needed from the INSTALL file at
>>>>>> http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce/INSTALL.
>>>>>> Reproducing some of those contents here for the short-term lookup.
>>>>>>
>>>>>> ------------------------------------
>>>>>> Checking out source code
>>>>>> ------------------------------------
>>>>>> svn checkout http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce
>>>>>>
>>>>>> --------------------------
>>>>>> Directory structure
>>>>>> --------------------------
>>>>>>
>>>>>> trunk/
>>>>>> - hadoop-mapreduce ( was mapreduce before)
>>>>>>
>>>>>> trunk/hadoop-mapreduce - Classic code. JT/TT reside here
>>>>>> - build.xml
>>>>>> - src
>>>>>>
>>>>>> trunk/hadoop-mapreduce/ - New code related to yarn reside here.
>>>>>> - assembly
>>>>>> - pom.xml
>>>>>> - hadoop-mr-client
>>>>>> - hadoop-yarn - Yarn APIs, libraries, and server code
>>>>>> -- hadoop-yarn-api
>>>>>> -- hadoop-yarn-common
>>>>>> -- hadoop-yarn-server - Server code, ResourceManager, NodeManager,
>>>>>> server libraries and tests.
>>>>>> --- hadoop-yarn-server-common
>>>>>> --- hadoop-yarn-server-nodemanager
>>>>>> --- hadoop-yarn-server-resourcemanager
>>>>>> --- hadoop-yarn-server-tests
>>>>>> - hadoop-mr-client - MapReduce server and client code
>>>>>> -- hadoop-mapreduce-client-app
>>>>>> -- hadoop-mapreduce-client-core
>>>>>> -- hadoop-mapreduce-client-jobclient
>>>>>> -- hadoop-mapreduce-client-common
>>>>>> -- hadoop-mapreduce-client-hs
>>>>>> -- hadoop-mapreduce-client-shuffle
>>>>>>
>>>>>> -----------
>>>>>> Building
>>>>>> -----------
>>>>>> Building yarn code and install into the local maven cache.
>>>>>> - mvn clean install
>>>>>> - In case you want to skip the tests run: mvn clean install -DskipTests
>>>>>>
>>>>>> Building classic code once yarn code is built.
>>>>>> - ant veryclean jar jar-test -Dresolvers=internal
>>>>>>
>>>>>> ----------
>>>>>> Eclipse
>>>>>> -----------
>>>>>> 1) For hacking on the new yarn+MR code in eclipse, you should run
>>>>>> "mvn eclipse:eclipse" and then import the checked out source root as a
>>>>>> maven project.
>>>>>> 2) For developing on classic JT/TT code, running "ant eclipse" and
>>>>>> importing as java project should continue to work.
>>>>>>
>>>>>> Hope that helps. If you run into issues, please send an email or
>>>>>> create a JIRA issue.
>>>>>>
>>>>>> Thanks,
>>>>>> +Vinod
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Todd Lipcon
>>>> Software Engineer, Cloudera
>>>>
>>
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Notes for working on mapreduce trunk after the MR-279 merge.
Posted by Todd Lipcon <to...@cloudera.com>.
Let me ping Jukka and see if he has any idea why it's not auto-updating anymore.
-Todd
On Fri, Aug 19, 2011 at 10:26 AM, Mahadev Konar <ma...@hortonworks.com> wrote:
> I am seeing this issue again. Todd, do we need to fix something else?
> I committed 2 patches one yest and one today and I cant get it on the git branch. Something off with git pulling from svn.
>
> Any other fix we need to do?
>
>
> thanks
> mahadev
> On Aug 18, 2011, at 10:05 AM, Vinod Kumar Vavilapalli wrote:
>
>> Thanks Todd!
>>
>> On Thu, Aug 18, 2011 at 10:27 PM, Todd Lipcon <to...@cloudera.com> wrote:
>>> I'm kicking the git mirror now... for some reason, on merges, it often
>>> takes several commits before it fully syncs up. I have access to
>>> manually poke it into going.
>>>
>>> -Todd
>>>
>>> On Thu, Aug 18, 2011 at 8:54 AM, Vinod Kumar Vavilapalli
>>> <vi...@hortonworks.com> wrote:
>>>> yes.. I also observed this. Looks like the mirrors are setup to use
>>>> the hardcoded path to trunk/mapreduce which changed now to
>>>> trunk/hadoop-mapreduce.
>>>>
>>>> Will see and get the right people to fix it.
>>>>
>>>> Thanks,
>>>> +Vinod
>>>>
>>>> On Thu, Aug 18, 2011 at 9:16 PM, Robert Evans <ev...@yahoo-inc.com> wrote:
>>>>> It looks like git has not seen the changes yet, even though the last change was over 90 mins ago. Is there any way to kick git to pull in the changes sooner so I can rebase?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Bobby Evans
>>>>>
>>>>> On 8/18/11 7:49 AM, "Vinod Kumar Vavilapalli" <vi...@hortonworks.com> wrote:
>>>>>
>>>>> MR-279 branch is merged into mapreduce trunk and this changes things a
>>>>> bit for developing on mapreduce.
>>>>>
>>>>> You can get all the help that is needed from the INSTALL file at
>>>>> http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce/INSTALL.
>>>>> Reproducing some of those contents here for the short-term lookup.
>>>>>
>>>>> ------------------------------------
>>>>> Checking out source code
>>>>> ------------------------------------
>>>>> svn checkout http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce
>>>>>
>>>>> --------------------------
>>>>> Directory structure
>>>>> --------------------------
>>>>>
>>>>> trunk/
>>>>> - hadoop-mapreduce ( was mapreduce before)
>>>>>
>>>>> trunk/hadoop-mapreduce - Classic code. JT/TT reside here
>>>>> - build.xml
>>>>> - src
>>>>>
>>>>> trunk/hadoop-mapreduce/ - New code related to yarn reside here.
>>>>> - assembly
>>>>> - pom.xml
>>>>> - hadoop-mr-client
>>>>> - hadoop-yarn - Yarn APIs, libraries, and server code
>>>>> -- hadoop-yarn-api
>>>>> -- hadoop-yarn-common
>>>>> -- hadoop-yarn-server - Server code, ResourceManager, NodeManager,
>>>>> server libraries and tests.
>>>>> --- hadoop-yarn-server-common
>>>>> --- hadoop-yarn-server-nodemanager
>>>>> --- hadoop-yarn-server-resourcemanager
>>>>> --- hadoop-yarn-server-tests
>>>>> - hadoop-mr-client - MapReduce server and client code
>>>>> -- hadoop-mapreduce-client-app
>>>>> -- hadoop-mapreduce-client-core
>>>>> -- hadoop-mapreduce-client-jobclient
>>>>> -- hadoop-mapreduce-client-common
>>>>> -- hadoop-mapreduce-client-hs
>>>>> -- hadoop-mapreduce-client-shuffle
>>>>>
>>>>> -----------
>>>>> Building
>>>>> -----------
>>>>> Building yarn code and install into the local maven cache.
>>>>> - mvn clean install
>>>>> - In case you want to skip the tests run: mvn clean install -DskipTests
>>>>>
>>>>> Building classic code once yarn code is built.
>>>>> - ant veryclean jar jar-test -Dresolvers=internal
>>>>>
>>>>> ----------
>>>>> Eclipse
>>>>> -----------
>>>>> 1) For hacking on the new yarn+MR code in eclipse, you should run
>>>>> "mvn eclipse:eclipse" and then import the checked out source root as a
>>>>> maven project.
>>>>> 2) For developing on classic JT/TT code, running "ant eclipse" and
>>>>> importing as java project should continue to work.
>>>>>
>>>>> Hope that helps. If you run into issues, please send an email or
>>>>> create a JIRA issue.
>>>>>
>>>>> Thanks,
>>>>> +Vinod
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>>>
>
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Notes for working on mapreduce trunk after the MR-279 merge.
Posted by Mahadev Konar <ma...@hortonworks.com>.
I am seeing this issue again. Todd, do we need to fix something else?
I committed 2 patches one yest and one today and I cant get it on the git branch. Something off with git pulling from svn.
Any other fix we need to do?
thanks
mahadev
On Aug 18, 2011, at 10:05 AM, Vinod Kumar Vavilapalli wrote:
> Thanks Todd!
>
> On Thu, Aug 18, 2011 at 10:27 PM, Todd Lipcon <to...@cloudera.com> wrote:
>> I'm kicking the git mirror now... for some reason, on merges, it often
>> takes several commits before it fully syncs up. I have access to
>> manually poke it into going.
>>
>> -Todd
>>
>> On Thu, Aug 18, 2011 at 8:54 AM, Vinod Kumar Vavilapalli
>> <vi...@hortonworks.com> wrote:
>>> yes.. I also observed this. Looks like the mirrors are setup to use
>>> the hardcoded path to trunk/mapreduce which changed now to
>>> trunk/hadoop-mapreduce.
>>>
>>> Will see and get the right people to fix it.
>>>
>>> Thanks,
>>> +Vinod
>>>
>>> On Thu, Aug 18, 2011 at 9:16 PM, Robert Evans <ev...@yahoo-inc.com> wrote:
>>>> It looks like git has not seen the changes yet, even though the last change was over 90 mins ago. Is there any way to kick git to pull in the changes sooner so I can rebase?
>>>>
>>>> Thanks,
>>>>
>>>> Bobby Evans
>>>>
>>>> On 8/18/11 7:49 AM, "Vinod Kumar Vavilapalli" <vi...@hortonworks.com> wrote:
>>>>
>>>> MR-279 branch is merged into mapreduce trunk and this changes things a
>>>> bit for developing on mapreduce.
>>>>
>>>> You can get all the help that is needed from the INSTALL file at
>>>> http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce/INSTALL.
>>>> Reproducing some of those contents here for the short-term lookup.
>>>>
>>>> ------------------------------------
>>>> Checking out source code
>>>> ------------------------------------
>>>> svn checkout http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce
>>>>
>>>> --------------------------
>>>> Directory structure
>>>> --------------------------
>>>>
>>>> trunk/
>>>> - hadoop-mapreduce ( was mapreduce before)
>>>>
>>>> trunk/hadoop-mapreduce - Classic code. JT/TT reside here
>>>> - build.xml
>>>> - src
>>>>
>>>> trunk/hadoop-mapreduce/ - New code related to yarn reside here.
>>>> - assembly
>>>> - pom.xml
>>>> - hadoop-mr-client
>>>> - hadoop-yarn - Yarn APIs, libraries, and server code
>>>> -- hadoop-yarn-api
>>>> -- hadoop-yarn-common
>>>> -- hadoop-yarn-server - Server code, ResourceManager, NodeManager,
>>>> server libraries and tests.
>>>> --- hadoop-yarn-server-common
>>>> --- hadoop-yarn-server-nodemanager
>>>> --- hadoop-yarn-server-resourcemanager
>>>> --- hadoop-yarn-server-tests
>>>> - hadoop-mr-client - MapReduce server and client code
>>>> -- hadoop-mapreduce-client-app
>>>> -- hadoop-mapreduce-client-core
>>>> -- hadoop-mapreduce-client-jobclient
>>>> -- hadoop-mapreduce-client-common
>>>> -- hadoop-mapreduce-client-hs
>>>> -- hadoop-mapreduce-client-shuffle
>>>>
>>>> -----------
>>>> Building
>>>> -----------
>>>> Building yarn code and install into the local maven cache.
>>>> - mvn clean install
>>>> - In case you want to skip the tests run: mvn clean install -DskipTests
>>>>
>>>> Building classic code once yarn code is built.
>>>> - ant veryclean jar jar-test -Dresolvers=internal
>>>>
>>>> ----------
>>>> Eclipse
>>>> -----------
>>>> 1) For hacking on the new yarn+MR code in eclipse, you should run
>>>> "mvn eclipse:eclipse" and then import the checked out source root as a
>>>> maven project.
>>>> 2) For developing on classic JT/TT code, running "ant eclipse" and
>>>> importing as java project should continue to work.
>>>>
>>>> Hope that helps. If you run into issues, please send an email or
>>>> create a JIRA issue.
>>>>
>>>> Thanks,
>>>> +Vinod
>>>>
>>>>
>>>
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
Re: Notes for working on mapreduce trunk after the MR-279 merge.
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
Thanks Todd!
On Thu, Aug 18, 2011 at 10:27 PM, Todd Lipcon <to...@cloudera.com> wrote:
> I'm kicking the git mirror now... for some reason, on merges, it often
> takes several commits before it fully syncs up. I have access to
> manually poke it into going.
>
> -Todd
>
> On Thu, Aug 18, 2011 at 8:54 AM, Vinod Kumar Vavilapalli
> <vi...@hortonworks.com> wrote:
>> yes.. I also observed this. Looks like the mirrors are setup to use
>> the hardcoded path to trunk/mapreduce which changed now to
>> trunk/hadoop-mapreduce.
>>
>> Will see and get the right people to fix it.
>>
>> Thanks,
>> +Vinod
>>
>> On Thu, Aug 18, 2011 at 9:16 PM, Robert Evans <ev...@yahoo-inc.com> wrote:
>>> It looks like git has not seen the changes yet, even though the last change was over 90 mins ago. Is there any way to kick git to pull in the changes sooner so I can rebase?
>>>
>>> Thanks,
>>>
>>> Bobby Evans
>>>
>>> On 8/18/11 7:49 AM, "Vinod Kumar Vavilapalli" <vi...@hortonworks.com> wrote:
>>>
>>> MR-279 branch is merged into mapreduce trunk and this changes things a
>>> bit for developing on mapreduce.
>>>
>>> You can get all the help that is needed from the INSTALL file at
>>> http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce/INSTALL.
>>> Reproducing some of those contents here for the short-term lookup.
>>>
>>> ------------------------------------
>>> Checking out source code
>>> ------------------------------------
>>> svn checkout http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce
>>>
>>> --------------------------
>>> Directory structure
>>> --------------------------
>>>
>>> trunk/
>>> - hadoop-mapreduce ( was mapreduce before)
>>>
>>> trunk/hadoop-mapreduce - Classic code. JT/TT reside here
>>> - build.xml
>>> - src
>>>
>>> trunk/hadoop-mapreduce/ - New code related to yarn reside here.
>>> - assembly
>>> - pom.xml
>>> - hadoop-mr-client
>>> - hadoop-yarn - Yarn APIs, libraries, and server code
>>> -- hadoop-yarn-api
>>> -- hadoop-yarn-common
>>> -- hadoop-yarn-server - Server code, ResourceManager, NodeManager,
>>> server libraries and tests.
>>> --- hadoop-yarn-server-common
>>> --- hadoop-yarn-server-nodemanager
>>> --- hadoop-yarn-server-resourcemanager
>>> --- hadoop-yarn-server-tests
>>> - hadoop-mr-client - MapReduce server and client code
>>> -- hadoop-mapreduce-client-app
>>> -- hadoop-mapreduce-client-core
>>> -- hadoop-mapreduce-client-jobclient
>>> -- hadoop-mapreduce-client-common
>>> -- hadoop-mapreduce-client-hs
>>> -- hadoop-mapreduce-client-shuffle
>>>
>>> -----------
>>> Building
>>> -----------
>>> Building yarn code and install into the local maven cache.
>>> - mvn clean install
>>> - In case you want to skip the tests run: mvn clean install -DskipTests
>>>
>>> Building classic code once yarn code is built.
>>> - ant veryclean jar jar-test -Dresolvers=internal
>>>
>>> ----------
>>> Eclipse
>>> -----------
>>> 1) For hacking on the new yarn+MR code in eclipse, you should run
>>> "mvn eclipse:eclipse" and then import the checked out source root as a
>>> maven project.
>>> 2) For developing on classic JT/TT code, running "ant eclipse" and
>>> importing as java project should continue to work.
>>>
>>> Hope that helps. If you run into issues, please send an email or
>>> create a JIRA issue.
>>>
>>> Thanks,
>>> +Vinod
>>>
>>>
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
Re: Notes for working on mapreduce trunk after the MR-279 merge.
Posted by Todd Lipcon <to...@cloudera.com>.
I'm kicking the git mirror now... for some reason, on merges, it often
takes several commits before it fully syncs up. I have access to
manually poke it into going.
-Todd
On Thu, Aug 18, 2011 at 8:54 AM, Vinod Kumar Vavilapalli
<vi...@hortonworks.com> wrote:
> yes.. I also observed this. Looks like the mirrors are setup to use
> the hardcoded path to trunk/mapreduce which changed now to
> trunk/hadoop-mapreduce.
>
> Will see and get the right people to fix it.
>
> Thanks,
> +Vinod
>
> On Thu, Aug 18, 2011 at 9:16 PM, Robert Evans <ev...@yahoo-inc.com> wrote:
>> It looks like git has not seen the changes yet, even though the last change was over 90 mins ago. Is there any way to kick git to pull in the changes sooner so I can rebase?
>>
>> Thanks,
>>
>> Bobby Evans
>>
>> On 8/18/11 7:49 AM, "Vinod Kumar Vavilapalli" <vi...@hortonworks.com> wrote:
>>
>> MR-279 branch is merged into mapreduce trunk and this changes things a
>> bit for developing on mapreduce.
>>
>> You can get all the help that is needed from the INSTALL file at
>> http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce/INSTALL.
>> Reproducing some of those contents here for the short-term lookup.
>>
>> ------------------------------------
>> Checking out source code
>> ------------------------------------
>> svn checkout http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce
>>
>> --------------------------
>> Directory structure
>> --------------------------
>>
>> trunk/
>> - hadoop-mapreduce ( was mapreduce before)
>>
>> trunk/hadoop-mapreduce - Classic code. JT/TT reside here
>> - build.xml
>> - src
>>
>> trunk/hadoop-mapreduce/ - New code related to yarn reside here.
>> - assembly
>> - pom.xml
>> - hadoop-mr-client
>> - hadoop-yarn - Yarn APIs, libraries, and server code
>> -- hadoop-yarn-api
>> -- hadoop-yarn-common
>> -- hadoop-yarn-server - Server code, ResourceManager, NodeManager,
>> server libraries and tests.
>> --- hadoop-yarn-server-common
>> --- hadoop-yarn-server-nodemanager
>> --- hadoop-yarn-server-resourcemanager
>> --- hadoop-yarn-server-tests
>> - hadoop-mr-client - MapReduce server and client code
>> -- hadoop-mapreduce-client-app
>> -- hadoop-mapreduce-client-core
>> -- hadoop-mapreduce-client-jobclient
>> -- hadoop-mapreduce-client-common
>> -- hadoop-mapreduce-client-hs
>> -- hadoop-mapreduce-client-shuffle
>>
>> -----------
>> Building
>> -----------
>> Building yarn code and install into the local maven cache.
>> - mvn clean install
>> - In case you want to skip the tests run: mvn clean install -DskipTests
>>
>> Building classic code once yarn code is built.
>> - ant veryclean jar jar-test -Dresolvers=internal
>>
>> ----------
>> Eclipse
>> -----------
>> 1) For hacking on the new yarn+MR code in eclipse, you should run
>> "mvn eclipse:eclipse" and then import the checked out source root as a
>> maven project.
>> 2) For developing on classic JT/TT code, running "ant eclipse" and
>> importing as java project should continue to work.
>>
>> Hope that helps. If you run into issues, please send an email or
>> create a JIRA issue.
>>
>> Thanks,
>> +Vinod
>>
>>
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Notes for working on mapreduce trunk after the MR-279 merge.
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
yes.. I also observed this. Looks like the mirrors are setup to use
the hardcoded path to trunk/mapreduce which changed now to
trunk/hadoop-mapreduce.
Will see and get the right people to fix it.
Thanks,
+Vinod
On Thu, Aug 18, 2011 at 9:16 PM, Robert Evans <ev...@yahoo-inc.com> wrote:
> It looks like git has not seen the changes yet, even though the last change was over 90 mins ago. Is there any way to kick git to pull in the changes sooner so I can rebase?
>
> Thanks,
>
> Bobby Evans
>
> On 8/18/11 7:49 AM, "Vinod Kumar Vavilapalli" <vi...@hortonworks.com> wrote:
>
> MR-279 branch is merged into mapreduce trunk and this changes things a
> bit for developing on mapreduce.
>
> You can get all the help that is needed from the INSTALL file at
> http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce/INSTALL.
> Reproducing some of those contents here for the short-term lookup.
>
> ------------------------------------
> Checking out source code
> ------------------------------------
> svn checkout http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce
>
> --------------------------
> Directory structure
> --------------------------
>
> trunk/
> - hadoop-mapreduce ( was mapreduce before)
>
> trunk/hadoop-mapreduce - Classic code. JT/TT reside here
> - build.xml
> - src
>
> trunk/hadoop-mapreduce/ - New code related to yarn reside here.
> - assembly
> - pom.xml
> - hadoop-mr-client
> - hadoop-yarn - Yarn APIs, libraries, and server code
> -- hadoop-yarn-api
> -- hadoop-yarn-common
> -- hadoop-yarn-server - Server code, ResourceManager, NodeManager,
> server libraries and tests.
> --- hadoop-yarn-server-common
> --- hadoop-yarn-server-nodemanager
> --- hadoop-yarn-server-resourcemanager
> --- hadoop-yarn-server-tests
> - hadoop-mr-client - MapReduce server and client code
> -- hadoop-mapreduce-client-app
> -- hadoop-mapreduce-client-core
> -- hadoop-mapreduce-client-jobclient
> -- hadoop-mapreduce-client-common
> -- hadoop-mapreduce-client-hs
> -- hadoop-mapreduce-client-shuffle
>
> -----------
> Building
> -----------
> Building yarn code and install into the local maven cache.
> - mvn clean install
> - In case you want to skip the tests run: mvn clean install -DskipTests
>
> Building classic code once yarn code is built.
> - ant veryclean jar jar-test -Dresolvers=internal
>
> ----------
> Eclipse
> -----------
> 1) For hacking on the new yarn+MR code in eclipse, you should run
> "mvn eclipse:eclipse" and then import the checked out source root as a
> maven project.
> 2) For developing on classic JT/TT code, running "ant eclipse" and
> importing as java project should continue to work.
>
> Hope that helps. If you run into issues, please send an email or
> create a JIRA issue.
>
> Thanks,
> +Vinod
>
>
Re: Notes for working on mapreduce trunk after the MR-279 merge.
Posted by Thomas Graves <tg...@yahoo-inc.com>.
I just built and brought up mrv2 on a single node cluster. I was
successfully able to run wordcount from the mapreduce-examples after adding
the following config change to my mapred-site.xml. Without this change it
was still trying to contact the jobtracker.
<property>
<name> mapreduce.framework.name</name>
<value>yarn</value>
</property>
I'll file a jira and update the INSTALL doc.
Tom
On 8/18/11 7:49 AM, "Vinod Kumar Vavilapalli" <vi...@hortonworks.com>
wrote:
> MR-279 branch is merged into mapreduce trunk and this changes things a
> bit for developing on mapreduce.
>
> You can get all the help that is needed from the INSTALL file at
> http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce/INSTALL.
> Reproducing some of those contents here for the short-term lookup.
>
> ------------------------------------
> Checking out source code
> ------------------------------------
> svn checkout
> http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce
>
> --------------------------
> Directory structure
> --------------------------
>
> trunk/
> - hadoop-mapreduce ( was mapreduce before)
>
> trunk/hadoop-mapreduce - Classic code. JT/TT reside here
> - build.xml
> - src
>
> trunk/hadoop-mapreduce/ - New code related to yarn reside here.
> - assembly
> - pom.xml
> - hadoop-mr-client
> - hadoop-yarn - Yarn APIs, libraries, and server code
> -- hadoop-yarn-api
> -- hadoop-yarn-common
> -- hadoop-yarn-server - Server code, ResourceManager, NodeManager,
> server libraries and tests.
> --- hadoop-yarn-server-common
> --- hadoop-yarn-server-nodemanager
> --- hadoop-yarn-server-resourcemanager
> --- hadoop-yarn-server-tests
> - hadoop-mr-client - MapReduce server and client code
> -- hadoop-mapreduce-client-app
> -- hadoop-mapreduce-client-core
> -- hadoop-mapreduce-client-jobclient
> -- hadoop-mapreduce-client-common
> -- hadoop-mapreduce-client-hs
> -- hadoop-mapreduce-client-shuffle
>
> -----------
> Building
> -----------
> Building yarn code and install into the local maven cache.
> - mvn clean install
> - In case you want to skip the tests run: mvn clean install -DskipTests
>
> Building classic code once yarn code is built.
> - ant veryclean jar jar-test -Dresolvers=internal
>
> ----------
> Eclipse
> -----------
> 1) For hacking on the new yarn+MR code in eclipse, you should run
> "mvn eclipse:eclipse" and then import the checked out source root as a
> maven project.
> 2) For developing on classic JT/TT code, running "ant eclipse" and
> importing as java project should continue to work.
>
> Hope that helps. If you run into issues, please send an email or
> create a JIRA issue.
>
> Thanks,
> +Vinod
Re: Notes for working on mapreduce trunk after the MR-279 merge.
Posted by Robert Evans <ev...@yahoo-inc.com>.
It looks like git has not seen the changes yet, even though the last change was over 90 mins ago. Is there any way to kick git to pull in the changes sooner so I can rebase?
Thanks,
Bobby Evans
On 8/18/11 7:49 AM, "Vinod Kumar Vavilapalli" <vi...@hortonworks.com> wrote:
MR-279 branch is merged into mapreduce trunk and this changes things a
bit for developing on mapreduce.
You can get all the help that is needed from the INSTALL file at
http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce/INSTALL.
Reproducing some of those contents here for the short-term lookup.
------------------------------------
Checking out source code
------------------------------------
svn checkout http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce
--------------------------
Directory structure
--------------------------
trunk/
- hadoop-mapreduce ( was mapreduce before)
trunk/hadoop-mapreduce - Classic code. JT/TT reside here
- build.xml
- src
trunk/hadoop-mapreduce/ - New code related to yarn reside here.
- assembly
- pom.xml
- hadoop-mr-client
- hadoop-yarn - Yarn APIs, libraries, and server code
-- hadoop-yarn-api
-- hadoop-yarn-common
-- hadoop-yarn-server - Server code, ResourceManager, NodeManager,
server libraries and tests.
--- hadoop-yarn-server-common
--- hadoop-yarn-server-nodemanager
--- hadoop-yarn-server-resourcemanager
--- hadoop-yarn-server-tests
- hadoop-mr-client - MapReduce server and client code
-- hadoop-mapreduce-client-app
-- hadoop-mapreduce-client-core
-- hadoop-mapreduce-client-jobclient
-- hadoop-mapreduce-client-common
-- hadoop-mapreduce-client-hs
-- hadoop-mapreduce-client-shuffle
-----------
Building
-----------
Building yarn code and install into the local maven cache.
- mvn clean install
- In case you want to skip the tests run: mvn clean install -DskipTests
Building classic code once yarn code is built.
- ant veryclean jar jar-test -Dresolvers=internal
----------
Eclipse
-----------
1) For hacking on the new yarn+MR code in eclipse, you should run
"mvn eclipse:eclipse" and then import the checked out source root as a
maven project.
2) For developing on classic JT/TT code, running "ant eclipse" and
importing as java project should continue to work.
Hope that helps. If you run into issues, please send an email or
create a JIRA issue.
Thanks,
+Vinod