You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Prashant Sharma (Created) (JIRA)" <ji...@apache.org> on 2011/10/19 11:50:11 UTC
[jira] [Created] (MAPREDUCE-3211) Fetch failures if used ephemeral
port for property "mapreduce.shuffle.port".
Fetch failures if used ephemeral port for property "mapreduce.shuffle.port".
-----------------------------------------------------------------------------
Key: MAPREDUCE-3211
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3211
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: job submission, mrv2
Affects Versions: 0.24.0
Environment: Linux
Reporter: Prashant Sharma
Assignee: Prashant Sharma
Fix For: 0.24.0
To reproduce the bug.
Use the following property in mapred-site.xml
<configuration>
<property>
<name> mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name> mapreduce.shuffle.port</name>
<value>0</value>
</property>
</configuration>
and following in yarn-site.xml
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce.shuffle</value>
<description>shuffle service that needs to be set for Map Reduce to run </description>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
Then try to start the yarn daemons.
And submit the job. (which would eventually fail).
Also observe NM logs. It says shuffle handler service started at port 0; :P
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3211) Fetch failures if used
ephemeral port for property "mapreduce.shuffle.port".
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130490#comment-13130490 ]
Hadoop QA commented on MAPREDUCE-3211:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12499658/MAPREDUCE-3211.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 160 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1066//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1066//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1066//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1066//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1066//console
This message is automatically generated.
> Fetch failures if used ephemeral port for property "mapreduce.shuffle.port".
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3211
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3211
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, mrv2
> Affects Versions: 0.24.0
> Environment: Linux
> Reporter: Prashant Sharma
> Assignee: Prashant Sharma
> Labels: ShuffleHandler
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-3211.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> To reproduce the bug.
> Use the following property in mapred-site.xml
> <configuration>
> <property>
> <name> mapreduce.framework.name</name>
> <value>yarn</value>
> </property>
> <property>
> <name> mapreduce.shuffle.port</name>
> <value>0</value>
> </property>
> </configuration>
> and following in yarn-site.xml
> <property>
> <name>yarn.nodemanager.aux-services</name>
> <value>mapreduce.shuffle</value>
> <description>shuffle service that needs to be set for Map Reduce to run </description>
> </property>
> <property>
> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
> <value>org.apache.hadoop.mapred.ShuffleHandler</value>
> </property>
> Then try to start the yarn daemons.
> And submit the job. (which would eventually fail).
> Also observe NM logs. It says shuffle handler service started at port 0; :P
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3211) Fetch failures if used
ephemeral port for property "mapreduce.shuffle.port".
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130629#comment-13130629 ]
Hadoop QA commented on MAPREDUCE-3211:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12499677/MAPREDUCE-3211.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 160 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1068//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1068//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1068//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1068//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1068//console
This message is automatically generated.
> Fetch failures if used ephemeral port for property "mapreduce.shuffle.port".
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3211
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3211
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, mrv2
> Affects Versions: 0.24.0
> Environment: Linux
> Reporter: Prashant Sharma
> Assignee: Prashant Sharma
> Labels: ShuffleHandler
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-3211.patch, MAPREDUCE-3211.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> To reproduce the bug.
> Use the following property in mapred-site.xml
> <configuration>
> <property>
> <name> mapreduce.framework.name</name>
> <value>yarn</value>
> </property>
> <property>
> <name> mapreduce.shuffle.port</name>
> <value>0</value>
> </property>
> </configuration>
> and following in yarn-site.xml
> <property>
> <name>yarn.nodemanager.aux-services</name>
> <value>mapreduce.shuffle</value>
> <description>shuffle service that needs to be set for Map Reduce to run </description>
> </property>
> <property>
> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
> <value>org.apache.hadoop.mapred.ShuffleHandler</value>
> </property>
> Then try to start the yarn daemons.
> And submit the job. (which would eventually fail).
> Also observe NM logs. It says shuffle handler service started at port 0; :P
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3211) Fetch failures if used ephemeral
port for property "mapreduce.shuffle.port".
Posted by "Prashant Sharma (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prashant Sharma resolved MAPREDUCE-3211.
----------------------------------------
Resolution: Duplicate
Hadoop Flags: Reviewed
MAPREDUCE-2986
> Fetch failures if used ephemeral port for property "mapreduce.shuffle.port".
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3211
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3211
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, mrv2
> Affects Versions: 0.24.0
> Environment: Linux
> Reporter: Prashant Sharma
> Assignee: Prashant Sharma
> Labels: ShuffleHandler
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-3211.patch, MAPREDUCE-3211.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> To reproduce the bug.
> Use the following property in mapred-site.xml
> <configuration>
> <property>
> <name> mapreduce.framework.name</name>
> <value>yarn</value>
> </property>
> <property>
> <name> mapreduce.shuffle.port</name>
> <value>0</value>
> </property>
> </configuration>
> and following in yarn-site.xml
> <property>
> <name>yarn.nodemanager.aux-services</name>
> <value>mapreduce.shuffle</value>
> <description>shuffle service that needs to be set for Map Reduce to run </description>
> </property>
> <property>
> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
> <value>org.apache.hadoop.mapred.ShuffleHandler</value>
> </property>
> Then try to start the yarn daemons.
> And submit the job. (which would eventually fail).
> Also observe NM logs. It says shuffle handler service started at port 0; :P
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3211) Fetch failures if used
ephemeral port for property "mapreduce.shuffle.port".
Posted by "Robert Joseph Evans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130644#comment-13130644 ]
Robert Joseph Evans commented on MAPREDUCE-3211:
------------------------------------------------
The change looks good to me. I also don't know what the findbugs is talking about. As for the tests you could update the some of the integration tests that use the MiniMRYarnCluster to use an ephemeral port.
> Fetch failures if used ephemeral port for property "mapreduce.shuffle.port".
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3211
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3211
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, mrv2
> Affects Versions: 0.24.0
> Environment: Linux
> Reporter: Prashant Sharma
> Assignee: Prashant Sharma
> Labels: ShuffleHandler
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-3211.patch, MAPREDUCE-3211.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> To reproduce the bug.
> Use the following property in mapred-site.xml
> <configuration>
> <property>
> <name> mapreduce.framework.name</name>
> <value>yarn</value>
> </property>
> <property>
> <name> mapreduce.shuffle.port</name>
> <value>0</value>
> </property>
> </configuration>
> and following in yarn-site.xml
> <property>
> <name>yarn.nodemanager.aux-services</name>
> <value>mapreduce.shuffle</value>
> <description>shuffle service that needs to be set for Map Reduce to run </description>
> </property>
> <property>
> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
> <value>org.apache.hadoop.mapred.ShuffleHandler</value>
> </property>
> Then try to start the yarn daemons.
> And submit the job. (which would eventually fail).
> Also observe NM logs. It says shuffle handler service started at port 0; :P
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3211) Fetch failures if used ephemeral
port for property "mapreduce.shuffle.port".
Posted by "Prashant Sharma (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prashant Sharma updated MAPREDUCE-3211:
---------------------------------------
Attachment: MAPREDUCE-3211.patch
I have done a quick and obvious fix. Please review.
> Fetch failures if used ephemeral port for property "mapreduce.shuffle.port".
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3211
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3211
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, mrv2
> Affects Versions: 0.24.0
> Environment: Linux
> Reporter: Prashant Sharma
> Assignee: Prashant Sharma
> Labels: ShuffleHandler
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-3211.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> To reproduce the bug.
> Use the following property in mapred-site.xml
> <configuration>
> <property>
> <name> mapreduce.framework.name</name>
> <value>yarn</value>
> </property>
> <property>
> <name> mapreduce.shuffle.port</name>
> <value>0</value>
> </property>
> </configuration>
> and following in yarn-site.xml
> <property>
> <name>yarn.nodemanager.aux-services</name>
> <value>mapreduce.shuffle</value>
> <description>shuffle service that needs to be set for Map Reduce to run </description>
> </property>
> <property>
> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
> <value>org.apache.hadoop.mapred.ShuffleHandler</value>
> </property>
> Then try to start the yarn daemons.
> And submit the job. (which would eventually fail).
> Also observe NM logs. It says shuffle handler service started at port 0; :P
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3211) Fetch failures if used ephemeral
port for property "mapreduce.shuffle.port".
Posted by "Prashant Sharma (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prashant Sharma updated MAPREDUCE-3211:
---------------------------------------
Attachment: MAPREDUCE-3211.patch
> Fetch failures if used ephemeral port for property "mapreduce.shuffle.port".
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3211
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3211
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, mrv2
> Affects Versions: 0.24.0
> Environment: Linux
> Reporter: Prashant Sharma
> Assignee: Prashant Sharma
> Labels: ShuffleHandler
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-3211.patch, MAPREDUCE-3211.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> To reproduce the bug.
> Use the following property in mapred-site.xml
> <configuration>
> <property>
> <name> mapreduce.framework.name</name>
> <value>yarn</value>
> </property>
> <property>
> <name> mapreduce.shuffle.port</name>
> <value>0</value>
> </property>
> </configuration>
> and following in yarn-site.xml
> <property>
> <name>yarn.nodemanager.aux-services</name>
> <value>mapreduce.shuffle</value>
> <description>shuffle service that needs to be set for Map Reduce to run </description>
> </property>
> <property>
> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
> <value>org.apache.hadoop.mapred.ShuffleHandler</value>
> </property>
> Then try to start the yarn daemons.
> And submit the job. (which would eventually fail).
> Also observe NM logs. It says shuffle handler service started at port 0; :P
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (MAPREDUCE-3211) Fetch failures if used
ephemeral port for property "mapreduce.shuffle.port".
Posted by "Prashant Sharma (Work started) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on MAPREDUCE-3211 started by Prashant Sharma.
> Fetch failures if used ephemeral port for property "mapreduce.shuffle.port".
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3211
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3211
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, mrv2
> Affects Versions: 0.24.0
> Environment: Linux
> Reporter: Prashant Sharma
> Assignee: Prashant Sharma
> Labels: ShuffleHandler
> Fix For: 0.24.0
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> To reproduce the bug.
> Use the following property in mapred-site.xml
> <configuration>
> <property>
> <name> mapreduce.framework.name</name>
> <value>yarn</value>
> </property>
> <property>
> <name> mapreduce.shuffle.port</name>
> <value>0</value>
> </property>
> </configuration>
> and following in yarn-site.xml
> <property>
> <name>yarn.nodemanager.aux-services</name>
> <value>mapreduce.shuffle</value>
> <description>shuffle service that needs to be set for Map Reduce to run </description>
> </property>
> <property>
> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
> <value>org.apache.hadoop.mapred.ShuffleHandler</value>
> </property>
> Then try to start the yarn daemons.
> And submit the job. (which would eventually fail).
> Also observe NM logs. It says shuffle handler service started at port 0; :P
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3211) Fetch failures if used ephemeral
port for property "mapreduce.shuffle.port".
Posted by "Prashant Sharma (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prashant Sharma updated MAPREDUCE-3211:
---------------------------------------
Status: Patch Available (was: In Progress)
> Fetch failures if used ephemeral port for property "mapreduce.shuffle.port".
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3211
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3211
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, mrv2
> Affects Versions: 0.24.0
> Environment: Linux
> Reporter: Prashant Sharma
> Assignee: Prashant Sharma
> Labels: ShuffleHandler
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-3211.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> To reproduce the bug.
> Use the following property in mapred-site.xml
> <configuration>
> <property>
> <name> mapreduce.framework.name</name>
> <value>yarn</value>
> </property>
> <property>
> <name> mapreduce.shuffle.port</name>
> <value>0</value>
> </property>
> </configuration>
> and following in yarn-site.xml
> <property>
> <name>yarn.nodemanager.aux-services</name>
> <value>mapreduce.shuffle</value>
> <description>shuffle service that needs to be set for Map Reduce to run </description>
> </property>
> <property>
> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
> <value>org.apache.hadoop.mapred.ShuffleHandler</value>
> </property>
> Then try to start the yarn daemons.
> And submit the job. (which would eventually fail).
> Also observe NM logs. It says shuffle handler service started at port 0; :P
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3211) Fetch failures if used
ephemeral port for property "mapreduce.shuffle.port".
Posted by "Prashant Sharma (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130495#comment-13130495 ]
Prashant Sharma commented on MAPREDUCE-3211:
--------------------------------------------
Well justification for no test.
Could not think of any.This is a configuration dependent bug.
Findbug warnings are not there in my code.
> Fetch failures if used ephemeral port for property "mapreduce.shuffle.port".
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3211
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3211
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, mrv2
> Affects Versions: 0.24.0
> Environment: Linux
> Reporter: Prashant Sharma
> Assignee: Prashant Sharma
> Labels: ShuffleHandler
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-3211.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> To reproduce the bug.
> Use the following property in mapred-site.xml
> <configuration>
> <property>
> <name> mapreduce.framework.name</name>
> <value>yarn</value>
> </property>
> <property>
> <name> mapreduce.shuffle.port</name>
> <value>0</value>
> </property>
> </configuration>
> and following in yarn-site.xml
> <property>
> <name>yarn.nodemanager.aux-services</name>
> <value>mapreduce.shuffle</value>
> <description>shuffle service that needs to be set for Map Reduce to run </description>
> </property>
> <property>
> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
> <value>org.apache.hadoop.mapred.ShuffleHandler</value>
> </property>
> Then try to start the yarn daemons.
> And submit the job. (which would eventually fail).
> Also observe NM logs. It says shuffle handler service started at port 0; :P
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3211) Fetch failures if used ephemeral
port for property "mapreduce.shuffle.port".
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3211:
-----------------------------------------------
Status: Open (was: Patch Available)
> Fetch failures if used ephemeral port for property "mapreduce.shuffle.port".
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3211
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3211
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, mrv2
> Affects Versions: 0.24.0
> Environment: Linux
> Reporter: Prashant Sharma
> Assignee: Prashant Sharma
> Labels: ShuffleHandler
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-3211.patch, MAPREDUCE-3211.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> To reproduce the bug.
> Use the following property in mapred-site.xml
> <configuration>
> <property>
> <name> mapreduce.framework.name</name>
> <value>yarn</value>
> </property>
> <property>
> <name> mapreduce.shuffle.port</name>
> <value>0</value>
> </property>
> </configuration>
> and following in yarn-site.xml
> <property>
> <name>yarn.nodemanager.aux-services</name>
> <value>mapreduce.shuffle</value>
> <description>shuffle service that needs to be set for Map Reduce to run </description>
> </property>
> <property>
> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
> <value>org.apache.hadoop.mapred.ShuffleHandler</value>
> </property>
> Then try to start the yarn daemons.
> And submit the job. (which would eventually fail).
> Also observe NM logs. It says shuffle handler service started at port 0; :P
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3211) Fetch failures if used
ephemeral port for property "mapreduce.shuffle.port".
Posted by "Vinod Kumar Vavilapalli (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130847#comment-13130847 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-3211:
----------------------------------------------------
This same bug is getting fixed as part of MAPREDUCE-2986, along with other ports. Mind closing this as duplicate?
> Fetch failures if used ephemeral port for property "mapreduce.shuffle.port".
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3211
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3211
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, mrv2
> Affects Versions: 0.24.0
> Environment: Linux
> Reporter: Prashant Sharma
> Assignee: Prashant Sharma
> Labels: ShuffleHandler
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-3211.patch, MAPREDUCE-3211.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> To reproduce the bug.
> Use the following property in mapred-site.xml
> <configuration>
> <property>
> <name> mapreduce.framework.name</name>
> <value>yarn</value>
> </property>
> <property>
> <name> mapreduce.shuffle.port</name>
> <value>0</value>
> </property>
> </configuration>
> and following in yarn-site.xml
> <property>
> <name>yarn.nodemanager.aux-services</name>
> <value>mapreduce.shuffle</value>
> <description>shuffle service that needs to be set for Map Reduce to run </description>
> </property>
> <property>
> <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
> <value>org.apache.hadoop.mapred.ShuffleHandler</value>
> </property>
> Then try to start the yarn daemons.
> And submit the job. (which would eventually fail).
> Also observe NM logs. It says shuffle handler service started at port 0; :P
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira