You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Siddharth Seth (Created) (JIRA)" <ji...@apache.org> on 2012/02/06 05:21:59 UTC
[jira] [Created] (MAPREDUCE-3815) Data Locality suffers if HDFS
returns IPs in getFileBlockLocations
Data Locality suffers if HDFS returns IPs in getFileBlockLocations
------------------------------------------------------------------
Key: MAPREDUCE-3815
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3815
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Priority: Critical
BlockLocation.getHosts() returns IP addresses occasionally. Data locality is affected - since the RM requires hostnames.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3815) Data Locality suffers if the AM
asks for containers using IPs instead of hostnames
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202858#comment-13202858 ]
Hudson commented on MAPREDUCE-3815:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #1683 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1683/])
MAPREDUCE-3815. Fixed MR AM to always use hostnames and never IPs when requesting containers so that scheduler can give off data local containers correctly. Contributed by Siddarth Seth.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241654
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java
> Data Locality suffers if the AM asks for containers using IPs instead of hostnames
> ----------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3815
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3815
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MR3815.txt
>
>
> BlockLocation.getHosts() returns IP addresses occasionally. Data locality is affected - since the RM requires hostnames.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3815) Data Locality suffers if HDFS
returns IPs in getFileBlockLocations
Posted by "Siddharth Seth (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201497#comment-13201497 ]
Siddharth Seth commented on MAPREDUCE-3815:
-------------------------------------------
Both - will create a separate hdfs jira. This one is for a check in MR for such situations.
> Data Locality suffers if HDFS returns IPs in getFileBlockLocations
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-3815
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3815
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
>
> BlockLocation.getHosts() returns IP addresses occasionally. Data locality is affected - since the RM requires hostnames.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3815) Data Locality suffers if the AM
asks for containers using IPs instead of hostnames
Posted by "Siddharth Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated MAPREDUCE-3815:
--------------------------------------
Attachment: MR3815.txt
> Data Locality suffers if the AM asks for containers using IPs instead of hostnames
> ----------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3815
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3815
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Attachments: MR3815.txt
>
>
> BlockLocation.getHosts() returns IP addresses occasionally. Data locality is affected - since the RM requires hostnames.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3815) Data Locality suffers if the AM
asks for containers using IPs instead of hostnames
Posted by "Siddharth Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated MAPREDUCE-3815:
--------------------------------------
Status: Patch Available (was: Open)
> Data Locality suffers if the AM asks for containers using IPs instead of hostnames
> ----------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3815
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3815
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Attachments: MR3815.txt
>
>
> BlockLocation.getHosts() returns IP addresses occasionally. Data locality is affected - since the RM requires hostnames.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3815) Data Locality suffers if the AM
asks for containers using IPs instead of hostnames
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202757#comment-13202757 ]
Hadoop QA commented on MAPREDUCE-3815:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12513668/MR3815.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.mapred.TestJobCounters
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1811//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1811//console
This message is automatically generated.
> Data Locality suffers if the AM asks for containers using IPs instead of hostnames
> ----------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3815
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3815
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Attachments: MR3815.txt
>
>
> BlockLocation.getHosts() returns IP addresses occasionally. Data locality is affected - since the RM requires hostnames.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3815) Data Locality suffers if the AM
asks for containers using IPs instead of hostnames
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202893#comment-13202893 ]
Hudson commented on MAPREDUCE-3815:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1695 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1695/])
MAPREDUCE-3815. Fixed MR AM to always use hostnames and never IPs when requesting containers so that scheduler can give off data local containers correctly. Contributed by Siddarth Seth.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241654
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java
> Data Locality suffers if the AM asks for containers using IPs instead of hostnames
> ----------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3815
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3815
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MR3815.txt
>
>
> BlockLocation.getHosts() returns IP addresses occasionally. Data locality is affected - since the RM requires hostnames.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3815) Data Locality suffers if the AM
asks for containers using IPs instead of hostnames
Posted by "Siddharth Seth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated MAPREDUCE-3815:
--------------------------------------
Summary: Data Locality suffers if the AM asks for containers using IPs instead of hostnames (was: Data Locality suffers if HDFS returns IPs in getFileBlockLocations)
> Data Locality suffers if the AM asks for containers using IPs instead of hostnames
> ----------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3815
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3815
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Attachments: MR3815.txt
>
>
> BlockLocation.getHosts() returns IP addresses occasionally. Data locality is affected - since the RM requires hostnames.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3815) Data Locality suffers if HDFS
returns IPs in getFileBlockLocations
Posted by "Siddharth Seth (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth reassigned MAPREDUCE-3815:
-----------------------------------------
Assignee: Siddharth Seth
> Data Locality suffers if HDFS returns IPs in getFileBlockLocations
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-3815
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3815
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
>
> BlockLocation.getHosts() returns IP addresses occasionally. Data locality is affected - since the RM requires hostnames.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3815) Data Locality suffers if the AM
asks for containers using IPs instead of hostnames
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203547#comment-13203547 ]
Hudson commented on MAPREDUCE-3815:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #163 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/163/])
MAPREDUCE-3815. Fixed MR AM to always use hostnames and never IPs when requesting containers so that scheduler can give off data local containers correctly. Contributed by Siddarth Seth.
svn merge --ignore-ancestry -c 1241654 ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241655
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java
> Data Locality suffers if the AM asks for containers using IPs instead of hostnames
> ----------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3815
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3815
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MR3815.txt
>
>
> BlockLocation.getHosts() returns IP addresses occasionally. Data locality is affected - since the RM requires hostnames.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3815) Data Locality suffers if HDFS
returns IPs in getFileBlockLocations
Posted by "Siddharth Seth (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202685#comment-13202685 ]
Siddharth Seth commented on MAPREDUCE-3815:
-------------------------------------------
Looked at this a little more.
This shows up when a split spans across multiple blocks. {{getFileBlockLocations}} always returns hostnames. In case of multiple blocks, mapred.FileInputFormat ends up using {{BlockLocations.getTopologyPaths}} instead of getFileBlockLocations - which returns an IP address.
Will open a MR / HDFS jira once I can find out how this API behaves in the 1.0 line. Anyone happen to know ?
Meanwhile, changing the description and posting a patch to have the AM resolve IPs if they show up.
> Data Locality suffers if HDFS returns IPs in getFileBlockLocations
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-3815
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3815
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Attachments: MR3815.txt
>
>
> BlockLocation.getHosts() returns IP addresses occasionally. Data locality is affected - since the RM requires hostnames.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3815) Data Locality suffers if the AM
asks for containers using IPs instead of hostnames
Posted by "Vinod Kumar Vavilapalli (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202754#comment-13202754 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-3815:
----------------------------------------------------
Looking at the patch for a review.
> Data Locality suffers if the AM asks for containers using IPs instead of hostnames
> ----------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3815
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3815
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Attachments: MR3815.txt
>
>
> BlockLocation.getHosts() returns IP addresses occasionally. Data locality is affected - since the RM requires hostnames.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3815) Data Locality suffers if the AM
asks for containers using IPs instead of hostnames
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203571#comment-13203571 ]
Hudson commented on MAPREDUCE-3815:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #185 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/185/])
MAPREDUCE-3815. Fixed MR AM to always use hostnames and never IPs when requesting containers so that scheduler can give off data local containers correctly. Contributed by Siddarth Seth.
svn merge --ignore-ancestry -c 1241654 ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241655
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java
> Data Locality suffers if the AM asks for containers using IPs instead of hostnames
> ----------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3815
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3815
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MR3815.txt
>
>
> BlockLocation.getHosts() returns IP addresses occasionally. Data locality is affected - since the RM requires hostnames.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3815) Data Locality suffers if the AM
asks for containers using IPs instead of hostnames
Posted by "Vinod Kumar Vavilapalli (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202847#comment-13202847 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-3815:
----------------------------------------------------
+1. This looks good.
TestJobCounters failure is tracked at MAPREDUCE-3822.
I am pushing this in.
> Data Locality suffers if the AM asks for containers using IPs instead of hostnames
> ----------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3815
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3815
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Attachments: MR3815.txt
>
>
> BlockLocation.getHosts() returns IP addresses occasionally. Data locality is affected - since the RM requires hostnames.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3815) Data Locality suffers if the AM
asks for containers using IPs instead of hostnames
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203526#comment-13203526 ]
Hudson commented on MAPREDUCE-3815:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #950 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/950/])
MAPREDUCE-3815. Fixed MR AM to always use hostnames and never IPs when requesting containers so that scheduler can give off data local containers correctly. Contributed by Siddarth Seth.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241654
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java
> Data Locality suffers if the AM asks for containers using IPs instead of hostnames
> ----------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3815
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3815
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MR3815.txt
>
>
> BlockLocation.getHosts() returns IP addresses occasionally. Data locality is affected - since the RM requires hostnames.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3815) Data Locality suffers if the AM
asks for containers using IPs instead of hostnames
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3815:
-----------------------------------------------
Resolution: Fixed
Fix Version/s: 0.23.1
Release Note: Fixed MR AM to always use hostnames and never IPs when requesting containers so that scheduler can give off data local containers correctly.
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I just committed this to trunk and branch-0.23. Thanks Sid!
> Data Locality suffers if the AM asks for containers using IPs instead of hostnames
> ----------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3815
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3815
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MR3815.txt
>
>
> BlockLocation.getHosts() returns IP addresses occasionally. Data locality is affected - since the RM requires hostnames.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3815) Data Locality suffers if the AM
asks for containers using IPs instead of hostnames
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202856#comment-13202856 ]
Hudson commented on MAPREDUCE-3815:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1756 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1756/])
MAPREDUCE-3815. Fixed MR AM to always use hostnames and never IPs when requesting containers so that scheduler can give off data local containers correctly. Contributed by Siddarth Seth.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241654
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java
> Data Locality suffers if the AM asks for containers using IPs instead of hostnames
> ----------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3815
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3815
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MR3815.txt
>
>
> BlockLocation.getHosts() returns IP addresses occasionally. Data locality is affected - since the RM requires hostnames.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3815) Data Locality suffers if HDFS
returns IPs in getFileBlockLocations
Posted by "Eli Collins (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201080#comment-13201080 ]
Eli Collins commented on MAPREDUCE-3815:
----------------------------------------
This should be an HDFS jira right?
> Data Locality suffers if HDFS returns IPs in getFileBlockLocations
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-3815
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3815
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
>
> BlockLocation.getHosts() returns IP addresses occasionally. Data locality is affected - since the RM requires hostnames.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3815) Data Locality suffers if the AM
asks for containers using IPs instead of hostnames
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202864#comment-13202864 ]
Hudson commented on MAPREDUCE-3815:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Commit #499 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/499/])
MAPREDUCE-3815. Fixed MR AM to always use hostnames and never IPs when requesting containers so that scheduler can give off data local containers correctly. Contributed by Siddarth Seth.
svn merge --ignore-ancestry -c 1241654 ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241655
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java
> Data Locality suffers if the AM asks for containers using IPs instead of hostnames
> ----------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3815
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3815
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MR3815.txt
>
>
> BlockLocation.getHosts() returns IP addresses occasionally. Data locality is affected - since the RM requires hostnames.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3815) Data Locality suffers if the AM
asks for containers using IPs instead of hostnames
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202894#comment-13202894 ]
Hudson commented on MAPREDUCE-3815:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Commit #516 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/516/])
MAPREDUCE-3815. Fixed MR AM to always use hostnames and never IPs when requesting containers so that scheduler can give off data local containers correctly. Contributed by Siddarth Seth.
svn merge --ignore-ancestry -c 1241654 ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241655
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java
> Data Locality suffers if the AM asks for containers using IPs instead of hostnames
> ----------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3815
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3815
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MR3815.txt
>
>
> BlockLocation.getHosts() returns IP addresses occasionally. Data locality is affected - since the RM requires hostnames.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3815) Data Locality suffers if the AM
asks for containers using IPs instead of hostnames
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13203596#comment-13203596 ]
Hudson commented on MAPREDUCE-3815:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #983 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/983/])
MAPREDUCE-3815. Fixed MR AM to always use hostnames and never IPs when requesting containers so that scheduler can give off data local containers correctly. Contributed by Siddarth Seth.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241654
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java
> Data Locality suffers if the AM asks for containers using IPs instead of hostnames
> ----------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3815
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3815
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MR3815.txt
>
>
> BlockLocation.getHosts() returns IP addresses occasionally. Data locality is affected - since the RM requires hostnames.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3815) Data Locality suffers if the AM
asks for containers using IPs instead of hostnames
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202857#comment-13202857 ]
Hudson commented on MAPREDUCE-3815:
-----------------------------------
Integrated in Hadoop-Common-0.23-Commit #509 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/509/])
MAPREDUCE-3815. Fixed MR AM to always use hostnames and never IPs when requesting containers so that scheduler can give off data local containers correctly. Contributed by Siddarth Seth.
svn merge --ignore-ancestry -c 1241654 ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1241655
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java
> Data Locality suffers if the AM asks for containers using IPs instead of hostnames
> ----------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3815
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3815
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MR3815.txt
>
>
> BlockLocation.getHosts() returns IP addresses occasionally. Data locality is affected - since the RM requires hostnames.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3815) Data Locality suffers if HDFS
returns IPs in getFileBlockLocations
Posted by "Siddharth Seth (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201498#comment-13201498 ]
Siddharth Seth commented on MAPREDUCE-3815:
-------------------------------------------
Both - will create a separate hdfs jira. This one is for a check in MR for such situations.
> Data Locality suffers if HDFS returns IPs in getFileBlockLocations
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-3815
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3815
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
> Priority: Critical
>
> BlockLocation.getHosts() returns IP addresses occasionally. Data locality is affected - since the RM requires hostnames.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira