You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Josh Wills (JIRA)" <ji...@apache.org> on 2011/07/06 03:52:16 UTC
[jira] [Created] (MAPREDUCE-2645) Updates to MRv2 INSTALL
documentation
Updates to MRv2 INSTALL documentation
-------------------------------------
Key: MAPREDUCE-2645
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2645
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Reporter: Josh Wills
Assignee: Josh Wills
Attachments: MAPREDUCE-2645.patch
There are a few issues w/the current INSTALL document for MRv2 that I came across when I attempted to get it running:
1) Correct the mvn arg for skipping tests,
2) Add a step to start the yarn historyserver,
3) Add instructions to explicitly build the examples jar file and specify the mapreduce.clientfactory.class.name parameter correctly for MRv2.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2645) Updates to MRv2 INSTALL
documentation
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060736#comment-13060736 ]
Josh Wills commented on MAPREDUCE-2645:
---------------------------------------
Thanks for the review Vinod- that was very helpful to me. I added the changes to the README file that you suggested. A couple of replies to your other comments:
1) Re: HADOOP_HDFS_HOME, the need to set it explicitly is in Step 8 of the INSTALL file.
2) Re: ShuffleHandler, is that only true for mapred.*-based mapreduces, or mapreduce.* based ones as well?
Thanks again!
> Updates to MRv2 INSTALL documentation
> -------------------------------------
>
> Key: MAPREDUCE-2645
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2645
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Josh Wills
> Assignee: Josh Wills
> Attachments: MAPREDUCE-2645.patch
>
>
> There are a few issues w/the current INSTALL document for MRv2 that I came across when I attempted to get it running:
> 1) Correct the mvn arg for skipping tests,
> 2) Add a step to start the yarn historyserver,
> 3) Add instructions to explicitly build the examples jar file and specify the mapreduce.clientfactory.class.name parameter correctly for MRv2.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2645) Updates to MRv2 INSTALL
documentation
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13061305#comment-13061305 ]
Josh Wills commented on MAPREDUCE-2645:
---------------------------------------
On Thu, Jul 7, 2011 at 6:02 AM, Vinod Kumar Vavilapalli (JIRA) <
Thanks man. :)
Cool- yeah, I didn't realize that. Will do going forward.
> Updates to MRv2 INSTALL documentation
> -------------------------------------
>
> Key: MAPREDUCE-2645
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2645
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Josh Wills
> Assignee: Josh Wills
> Attachments: MAPREDUCE-2645.patch
>
>
> There are a few issues w/the current INSTALL document for MRv2 that I came across when I attempted to get it running:
> 1) Correct the mvn arg for skipping tests,
> 2) Add a step to start the yarn historyserver,
> 3) Add instructions to explicitly build the examples jar file and specify the mapreduce.clientfactory.class.name parameter correctly for MRv2.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2645) Updates to MRv2 INSTALL
documentation
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13061268#comment-13061268 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-2645:
----------------------------------------------------
bq. 2) Re: ShuffleHandler, is that only true for mapred.-based mapreduces, or mapreduce. based ones as well?
Mapreduce is a user land library now and so ShuffleHandler has to specially configured on the NMs to be able to run mapreduce jobs which also have reduces.
Regarding the config item, yes it points to org.apache.hadoop.mapred.ShuffleHandler and not to mapreduce.
Never mind anyways. I made the change myself.
I noted that the earlier patch is missing. Did you remove it by chance? The custom around here is that we keep uploading updated patches without deleting the older ones - serves as a history of the development :)
> Updates to MRv2 INSTALL documentation
> -------------------------------------
>
> Key: MAPREDUCE-2645
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2645
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Josh Wills
> Assignee: Josh Wills
> Attachments: MAPREDUCE-2645.patch
>
>
> There are a few issues w/the current INSTALL document for MRv2 that I came across when I attempted to get it running:
> 1) Correct the mvn arg for skipping tests,
> 2) Add a step to start the yarn historyserver,
> 3) Add instructions to explicitly build the examples jar file and specify the mapreduce.clientfactory.class.name parameter correctly for MRv2.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2645) Updates to MRv2 INSTALL
documentation
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Wills updated MAPREDUCE-2645:
----------------------------------
Attachment: (was: MAPREDUCE-2645.patch)
> Updates to MRv2 INSTALL documentation
> -------------------------------------
>
> Key: MAPREDUCE-2645
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2645
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Josh Wills
> Assignee: Josh Wills
> Attachments: MAPREDUCE-2645.patch
>
>
> There are a few issues w/the current INSTALL document for MRv2 that I came across when I attempted to get it running:
> 1) Correct the mvn arg for skipping tests,
> 2) Add a step to start the yarn historyserver,
> 3) Add instructions to explicitly build the examples jar file and specify the mapreduce.clientfactory.class.name parameter correctly for MRv2.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2645) Updates to MRv2 INSTALL
documentation
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060312#comment-13060312 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-2645:
----------------------------------------------------
Josh,
-Dtest.skip=true skips tests altogether, even compilcation. -Dmaven.test.skip.exec=true OTOH will compile them at the least. I guess we can differentiate these?
I just did a review of the current INSTALL doc to cover anything else that is missing. Some more points I'd like get added:
- Required version for Protobuf library is proto 2.4.0 and above. I found that the compilation gets broken with 2.3 erring on the option "java_generate_equals_and_hash" as unknown.
- Before starting NM, we should set HADOOP_HDFS_HOME specifically. Without HDFS_HDFS_HOME, NM fails localization.
- For launching MR jobs, we also need these configured in yarn-site.xml
<property>
<name>nodemanager.auxiluary.services</name>
<value>mapreduce.shuffle</value>
</property>
<property>
<name>nodemanager.aux.service.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
When you try to run with security enabled, you will need more settings, but that's for later :)
Thanks for looking into this!
> Updates to MRv2 INSTALL documentation
> -------------------------------------
>
> Key: MAPREDUCE-2645
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2645
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Josh Wills
> Assignee: Josh Wills
> Attachments: MAPREDUCE-2645.patch
>
>
> There are a few issues w/the current INSTALL document for MRv2 that I came across when I attempted to get it running:
> 1) Correct the mvn arg for skipping tests,
> 2) Add a step to start the yarn historyserver,
> 3) Add instructions to explicitly build the examples jar file and specify the mapreduce.clientfactory.class.name parameter correctly for MRv2.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2645) Updates to MRv2 INSTALL
documentation
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli resolved MAPREDUCE-2645.
------------------------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
I just committed this. Thanks Josh!
> Updates to MRv2 INSTALL documentation
> -------------------------------------
>
> Key: MAPREDUCE-2645
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2645
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Josh Wills
> Assignee: Josh Wills
> Attachments: MAPREDUCE-2645.patch
>
>
> There are a few issues w/the current INSTALL document for MRv2 that I came across when I attempted to get it running:
> 1) Correct the mvn arg for skipping tests,
> 2) Add a step to start the yarn historyserver,
> 3) Add instructions to explicitly build the examples jar file and specify the mapreduce.clientfactory.class.name parameter correctly for MRv2.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2645) Updates to MRv2 INSTALL
documentation
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060549#comment-13060549 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-2645:
----------------------------------------------------
Yikes, just realised there is a README too in the yarn directory.
bq. Required version for Protobuf library is proto 2.4.0 and above. I found that the compilation gets broken with 2.3 erring on the option "java_generate_equals_and_hash" as unknown.
We can drop my comment above, the README is clear about this.
Will be glad if we can make the changes in the README file too.
- Maven plugin is no longer needed to be installed separately. We can knock those steps off.
- The entire section under "General instructions for starting Yarn and MapReduce" is stale and better explained by the instructions in the INSTALL file, it can be dumped altogether.
> Updates to MRv2 INSTALL documentation
> -------------------------------------
>
> Key: MAPREDUCE-2645
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2645
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Josh Wills
> Assignee: Josh Wills
> Attachments: MAPREDUCE-2645.patch
>
>
> There are a few issues w/the current INSTALL document for MRv2 that I came across when I attempted to get it running:
> 1) Correct the mvn arg for skipping tests,
> 2) Add a step to start the yarn historyserver,
> 3) Add instructions to explicitly build the examples jar file and specify the mapreduce.clientfactory.class.name parameter correctly for MRv2.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2645) Updates to MRv2 INSTALL
documentation
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Wills updated MAPREDUCE-2645:
----------------------------------
Attachment: MAPREDUCE-2645.patch
INSTALL doc fixes, yarn/README fixes.
> Updates to MRv2 INSTALL documentation
> -------------------------------------
>
> Key: MAPREDUCE-2645
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2645
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Josh Wills
> Assignee: Josh Wills
> Attachments: MAPREDUCE-2645.patch
>
>
> There are a few issues w/the current INSTALL document for MRv2 that I came across when I attempted to get it running:
> 1) Correct the mvn arg for skipping tests,
> 2) Add a step to start the yarn historyserver,
> 3) Add instructions to explicitly build the examples jar file and specify the mapreduce.clientfactory.class.name parameter correctly for MRv2.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2645) Updates to MRv2 INSTALL
documentation
Posted by "Josh Wills (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Wills updated MAPREDUCE-2645:
----------------------------------
Attachment: MAPREDUCE-2645.patch
> Updates to MRv2 INSTALL documentation
> -------------------------------------
>
> Key: MAPREDUCE-2645
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2645
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Josh Wills
> Assignee: Josh Wills
> Attachments: MAPREDUCE-2645.patch
>
>
> There are a few issues w/the current INSTALL document for MRv2 that I came across when I attempted to get it running:
> 1) Correct the mvn arg for skipping tests,
> 2) Add a step to start the yarn historyserver,
> 3) Add instructions to explicitly build the examples jar file and specify the mapreduce.clientfactory.class.name parameter correctly for MRv2.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira