You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Devaraj K (JIRA)" <ji...@apache.org> on 2011/09/12 17:14:09 UTC
[jira] [Created] (MAPREDUCE-2983) All tasks are failing due to
invalid shuffle port number
All tasks are failing due to invalid shuffle port number
--------------------------------------------------------
Key: MAPREDUCE-2983
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2983
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster
Affects Versions: 0.24.0
Reporter: Devaraj K
{code:xml}
2011-09-12 18:43:10,361 INFO mapreduce.Job (Job.java:printTaskEvents(1227)) - Task Id : attempt_1315831998314_0007_r_000000_0, Status : FAILED
Container launch failed for container_1315831998314_0007_01_000003 : java.lang.IllegalStateException: Invalid shuffle port number -1 returned for attempt_1315831998314_0007_r_000000_0
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:226)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2983) All tasks are failing due to
invalid shuffle port number
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103319#comment-13103319 ]
Arun C Murthy commented on MAPREDUCE-2983:
------------------------------------------
Devaraj - I think it's better to open a new jira. Thanks.
> All tasks are failing due to invalid shuffle port number
> --------------------------------------------------------
>
> Key: MAPREDUCE-2983
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2983
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
> Affects Versions: 0.24.0
> Reporter: Devaraj K
>
> {code:xml}
> 2011-09-12 18:43:10,361 INFO mapreduce.Job (Job.java:printTaskEvents(1227)) - Task Id : attempt_1315831998314_0007_r_000000_0, Status : FAILED
> Container launch failed for container_1315831998314_0007_01_000003 : java.lang.IllegalStateException: Invalid shuffle port number -1 returned for attempt_1315831998314_0007_r_000000_0
> at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:226)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2983) All tasks are failing due to
invalid shuffle port number
Posted by "Devaraj K (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj K resolved MAPREDUCE-2983.
----------------------------------
Resolution: Not A Problem
Thanks Vinod. It works fine.
> All tasks are failing due to invalid shuffle port number
> --------------------------------------------------------
>
> Key: MAPREDUCE-2983
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2983
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
> Affects Versions: 0.24.0
> Reporter: Devaraj K
>
> {code:xml}
> 2011-09-12 18:43:10,361 INFO mapreduce.Job (Job.java:printTaskEvents(1227)) - Task Id : attempt_1315831998314_0007_r_000000_0, Status : FAILED
> Container launch failed for container_1315831998314_0007_01_000003 : java.lang.IllegalStateException: Invalid shuffle port number -1 returned for attempt_1315831998314_0007_r_000000_0
> at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:226)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2983) All tasks are failing due to
invalid shuffle port number
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13102768#comment-13102768 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-2983:
----------------------------------------------------
This happens if the NodeManagers are not configured to run the mapreduce specific ShuffleService.
Set _yarn.nodemanager.aux-services_ to _mapreduce.shuffle_ and _yarn.nodemanager.aux-services.mapreduce.shuffle.class_ to _org.apache.hadoop.mapred.ShuffleHandler_ in your yarn-site.xml
This is already mentioned in the INSTALL file.
Please close this ticket if that works for you. Thanks!
> All tasks are failing due to invalid shuffle port number
> --------------------------------------------------------
>
> Key: MAPREDUCE-2983
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2983
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
> Affects Versions: 0.24.0
> Reporter: Devaraj K
>
> {code:xml}
> 2011-09-12 18:43:10,361 INFO mapreduce.Job (Job.java:printTaskEvents(1227)) - Task Id : attempt_1315831998314_0007_r_000000_0, Status : FAILED
> Container launch failed for container_1315831998314_0007_01_000003 : java.lang.IllegalStateException: Invalid shuffle port number -1 returned for attempt_1315831998314_0007_r_000000_0
> at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:226)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2983) All tasks are failing due to
invalid shuffle port number
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13102876#comment-13102876 ]
Todd Lipcon commented on MAPREDUCE-2983:
----------------------------------------
Could we make this an easier experience for new users by having the MR AM double check that the ShuffleService is configured at application submission time? It seems like, if we know we won't be able to shuffle, we should fail with an obvious message early, rather than getting all the way to running tasks.
> All tasks are failing due to invalid shuffle port number
> --------------------------------------------------------
>
> Key: MAPREDUCE-2983
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2983
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
> Affects Versions: 0.24.0
> Reporter: Devaraj K
>
> {code:xml}
> 2011-09-12 18:43:10,361 INFO mapreduce.Job (Job.java:printTaskEvents(1227)) - Task Id : attempt_1315831998314_0007_r_000000_0, Status : FAILED
> Container launch failed for container_1315831998314_0007_01_000003 : java.lang.IllegalStateException: Invalid shuffle port number -1 returned for attempt_1315831998314_0007_r_000000_0
> at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:226)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2983) All tasks are failing due to
invalid shuffle port number
Posted by "Devaraj K (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103317#comment-13103317 ]
Devaraj K commented on MAPREDUCE-2983:
--------------------------------------
It is good idea Tood.
@Arun : Can we reopen this issue for providing the patch?
> All tasks are failing due to invalid shuffle port number
> --------------------------------------------------------
>
> Key: MAPREDUCE-2983
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2983
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
> Affects Versions: 0.24.0
> Reporter: Devaraj K
>
> {code:xml}
> 2011-09-12 18:43:10,361 INFO mapreduce.Job (Job.java:printTaskEvents(1227)) - Task Id : attempt_1315831998314_0007_r_000000_0, Status : FAILED
> Container launch failed for container_1315831998314_0007_01_000003 : java.lang.IllegalStateException: Invalid shuffle port number -1 returned for attempt_1315831998314_0007_r_000000_0
> at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:226)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2983) All tasks are failing due to
invalid shuffle port number
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104377#comment-13104377 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-2983:
----------------------------------------------------
bq. Could we make this an easier experience for new users by having the MR AM double check that the ShuffleService is configured at application submission time?
This is a hack irrespective of whether we do this in the AM or the client.
Unfortunately even that is insufficient. Shuffle service is a configured auxiliary service on the NodeManagers. Auxiliary services are designed to be, well, auxiliary. For e.g. you can imagine only a subset of the cluster to have mapreduce ShuffleService and the rest not to have it. So, the correct way to do this is make the AM request containers on nodes which have the specified auxiliary service configured, basically this should be part of container-requests. Will file a ticket for this.
> All tasks are failing due to invalid shuffle port number
> --------------------------------------------------------
>
> Key: MAPREDUCE-2983
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2983
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
> Affects Versions: 0.24.0
> Reporter: Devaraj K
>
> {code:xml}
> 2011-09-12 18:43:10,361 INFO mapreduce.Job (Job.java:printTaskEvents(1227)) - Task Id : attempt_1315831998314_0007_r_000000_0, Status : FAILED
> Container launch failed for container_1315831998314_0007_01_000003 : java.lang.IllegalStateException: Invalid shuffle port number -1 returned for attempt_1315831998314_0007_r_000000_0
> at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:226)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2983) All tasks are failing due to
invalid shuffle port number
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13102892#comment-13102892 ]
Arun C Murthy commented on MAPREDUCE-2983:
------------------------------------------
Good idea Todd. Can you please file a jira? Better yet, a patch!
> All tasks are failing due to invalid shuffle port number
> --------------------------------------------------------
>
> Key: MAPREDUCE-2983
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2983
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
> Affects Versions: 0.24.0
> Reporter: Devaraj K
>
> {code:xml}
> 2011-09-12 18:43:10,361 INFO mapreduce.Job (Job.java:printTaskEvents(1227)) - Task Id : attempt_1315831998314_0007_r_000000_0, Status : FAILED
> Container launch failed for container_1315831998314_0007_01_000003 : java.lang.IllegalStateException: Invalid shuffle port number -1 returned for attempt_1315831998314_0007_r_000000_0
> at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:226)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira