You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Devaraj K (JIRA)" <ji...@apache.org> on 2011/09/12 17:14:09 UTC

[jira] [Created] (MAPREDUCE-2983) All tasks are failing due to invalid shuffle port number

All tasks are failing due to invalid shuffle port number
--------------------------------------------------------

                 Key: MAPREDUCE-2983
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2983
             Project: Hadoop Map/Reduce
          Issue Type: Bug
          Components: applicationmaster
    Affects Versions: 0.24.0
            Reporter: Devaraj K


{code:xml}
2011-09-12 18:43:10,361 INFO  mapreduce.Job (Job.java:printTaskEvents(1227)) - Task Id : attempt_1315831998314_0007_r_000000_0, Status : FAILED
Container launch failed for container_1315831998314_0007_01_000003 : java.lang.IllegalStateException: Invalid shuffle port number -1 returned for attempt_1315831998314_0007_r_000000_0
        at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:226)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)

{code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (MAPREDUCE-2983) All tasks are failing due to invalid shuffle port number

Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103319#comment-13103319 ] 

Arun C Murthy commented on MAPREDUCE-2983:
------------------------------------------

Devaraj - I think it's better to open a new jira. Thanks.

> All tasks are failing due to invalid shuffle port number
> --------------------------------------------------------
>
>                 Key: MAPREDUCE-2983
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2983
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: applicationmaster
>    Affects Versions: 0.24.0
>            Reporter: Devaraj K
>
> {code:xml}
> 2011-09-12 18:43:10,361 INFO  mapreduce.Job (Job.java:printTaskEvents(1227)) - Task Id : attempt_1315831998314_0007_r_000000_0, Status : FAILED
> Container launch failed for container_1315831998314_0007_01_000003 : java.lang.IllegalStateException: Invalid shuffle port number -1 returned for attempt_1315831998314_0007_r_000000_0
>         at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:226)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:619)
> {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Resolved] (MAPREDUCE-2983) All tasks are failing due to invalid shuffle port number

Posted by "Devaraj K (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj K resolved MAPREDUCE-2983.
----------------------------------

    Resolution: Not A Problem

Thanks Vinod. It works fine.

> All tasks are failing due to invalid shuffle port number
> --------------------------------------------------------
>
>                 Key: MAPREDUCE-2983
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2983
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: applicationmaster
>    Affects Versions: 0.24.0
>            Reporter: Devaraj K
>
> {code:xml}
> 2011-09-12 18:43:10,361 INFO  mapreduce.Job (Job.java:printTaskEvents(1227)) - Task Id : attempt_1315831998314_0007_r_000000_0, Status : FAILED
> Container launch failed for container_1315831998314_0007_01_000003 : java.lang.IllegalStateException: Invalid shuffle port number -1 returned for attempt_1315831998314_0007_r_000000_0
>         at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:226)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:619)
> {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (MAPREDUCE-2983) All tasks are failing due to invalid shuffle port number

Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13102768#comment-13102768 ] 

Vinod Kumar Vavilapalli commented on MAPREDUCE-2983:
----------------------------------------------------

This happens if the NodeManagers are not configured to run the mapreduce specific ShuffleService.

Set _yarn.nodemanager.aux-services_ to _mapreduce.shuffle_ and _yarn.nodemanager.aux-services.mapreduce.shuffle.class_ to _org.apache.hadoop.mapred.ShuffleHandler_ in your yarn-site.xml

This is already mentioned in the INSTALL file.

Please close this ticket if that works for you. Thanks!

> All tasks are failing due to invalid shuffle port number
> --------------------------------------------------------
>
>                 Key: MAPREDUCE-2983
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2983
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: applicationmaster
>    Affects Versions: 0.24.0
>            Reporter: Devaraj K
>
> {code:xml}
> 2011-09-12 18:43:10,361 INFO  mapreduce.Job (Job.java:printTaskEvents(1227)) - Task Id : attempt_1315831998314_0007_r_000000_0, Status : FAILED
> Container launch failed for container_1315831998314_0007_01_000003 : java.lang.IllegalStateException: Invalid shuffle port number -1 returned for attempt_1315831998314_0007_r_000000_0
>         at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:226)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:619)
> {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (MAPREDUCE-2983) All tasks are failing due to invalid shuffle port number

Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13102876#comment-13102876 ] 

Todd Lipcon commented on MAPREDUCE-2983:
----------------------------------------

Could we make this an easier experience for new users by having the MR AM double check that the ShuffleService is configured at application submission time? It seems like, if we know we won't be able to shuffle, we should fail with an obvious message early, rather than getting all the way to running tasks.

> All tasks are failing due to invalid shuffle port number
> --------------------------------------------------------
>
>                 Key: MAPREDUCE-2983
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2983
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: applicationmaster
>    Affects Versions: 0.24.0
>            Reporter: Devaraj K
>
> {code:xml}
> 2011-09-12 18:43:10,361 INFO  mapreduce.Job (Job.java:printTaskEvents(1227)) - Task Id : attempt_1315831998314_0007_r_000000_0, Status : FAILED
> Container launch failed for container_1315831998314_0007_01_000003 : java.lang.IllegalStateException: Invalid shuffle port number -1 returned for attempt_1315831998314_0007_r_000000_0
>         at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:226)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:619)
> {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (MAPREDUCE-2983) All tasks are failing due to invalid shuffle port number

Posted by "Devaraj K (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103317#comment-13103317 ] 

Devaraj K commented on MAPREDUCE-2983:
--------------------------------------

It is good idea Tood.

@Arun : Can we reopen this issue for providing the patch?

> All tasks are failing due to invalid shuffle port number
> --------------------------------------------------------
>
>                 Key: MAPREDUCE-2983
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2983
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: applicationmaster
>    Affects Versions: 0.24.0
>            Reporter: Devaraj K
>
> {code:xml}
> 2011-09-12 18:43:10,361 INFO  mapreduce.Job (Job.java:printTaskEvents(1227)) - Task Id : attempt_1315831998314_0007_r_000000_0, Status : FAILED
> Container launch failed for container_1315831998314_0007_01_000003 : java.lang.IllegalStateException: Invalid shuffle port number -1 returned for attempt_1315831998314_0007_r_000000_0
>         at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:226)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:619)
> {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (MAPREDUCE-2983) All tasks are failing due to invalid shuffle port number

Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104377#comment-13104377 ] 

Vinod Kumar Vavilapalli commented on MAPREDUCE-2983:
----------------------------------------------------

bq. Could we make this an easier experience for new users by having the MR AM double check that the ShuffleService is configured at application submission time? 
This is a hack irrespective of whether we do this in the AM or the client.

Unfortunately even that is insufficient. Shuffle service is a configured auxiliary service on the NodeManagers. Auxiliary services are designed to be, well, auxiliary. For e.g. you can imagine only a subset of the cluster to have mapreduce ShuffleService and the rest not to have it. So, the correct way to do this is make the AM request containers on nodes which have the specified auxiliary service configured, basically this should be part of container-requests. Will file a ticket for this.

> All tasks are failing due to invalid shuffle port number
> --------------------------------------------------------
>
>                 Key: MAPREDUCE-2983
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2983
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: applicationmaster
>    Affects Versions: 0.24.0
>            Reporter: Devaraj K
>
> {code:xml}
> 2011-09-12 18:43:10,361 INFO  mapreduce.Job (Job.java:printTaskEvents(1227)) - Task Id : attempt_1315831998314_0007_r_000000_0, Status : FAILED
> Container launch failed for container_1315831998314_0007_01_000003 : java.lang.IllegalStateException: Invalid shuffle port number -1 returned for attempt_1315831998314_0007_r_000000_0
>         at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:226)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:619)
> {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (MAPREDUCE-2983) All tasks are failing due to invalid shuffle port number

Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13102892#comment-13102892 ] 

Arun C Murthy commented on MAPREDUCE-2983:
------------------------------------------

Good idea Todd. Can you please file a jira? Better yet, a patch!

> All tasks are failing due to invalid shuffle port number
> --------------------------------------------------------
>
>                 Key: MAPREDUCE-2983
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2983
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: applicationmaster
>    Affects Versions: 0.24.0
>            Reporter: Devaraj K
>
> {code:xml}
> 2011-09-12 18:43:10,361 INFO  mapreduce.Job (Job.java:printTaskEvents(1227)) - Task Id : attempt_1315831998314_0007_r_000000_0, Status : FAILED
> Container launch failed for container_1315831998314_0007_01_000003 : java.lang.IllegalStateException: Invalid shuffle port number -1 returned for attempt_1315831998314_0007_r_000000_0
>         at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:226)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:619)
> {code}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira