[jira] [Commented] (ACCUMULO-2036) Mappers are not running locally

["Josh Elser (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ACCUMULO-2036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13861165#comment-13861165 ] 

Josh Elser commented on ACCUMULO-2036:
--------------------------------------

Looks like you were correct about fqdn, Keith. From hadoop-2.2.0, this gets called when the string looks like an IP, and the string is added otherwise.

{noformat}
  protected String resolveHost(String src) {
    String result = src; // Fallback in case of failure.
    try {
      InetAddress addr = InetAddress.getByName(src);
      result = addr.getHostName();
    } catch (UnknownHostException e) {
      LOG.warn("Failed to resolve address: " + src
          + ". Continuing to use the same.");
    }
    return result;
  }
{noformat}

Ok, dug down into the Java source and ultimately some man pages, but I found the information I want:

{noformat}
The domain name queries carried out by gethostbyname() and gethostbyaddr() use a combination of any or all of the name server named(8),  a  broken  out in /etc/host.conf.  The default action is to query named(8), followed by /etc/hosts.
{noformat}

I think that explains when/how Java will be returning the FQDN from the alias.

Just need to find again where Hadoop is assigning the mappers to TaskTrackers (or rather Map task to Container on NodeManager) to get the locality.

> Mappers are not running locally
> -------------------------------
>
>                 Key: ACCUMULO-2036
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-2036
>             Project: Accumulo
>          Issue Type: Bug
>            Reporter: Keith Turner
>            Assignee: Josh Elser
>             Fix For: 1.6.0
>
>
> I ran listscans in the Accumulo shell while running continuous verify on small cluster and almost not map task were running locally.
> I think ACCUMULO-1585 has broken mapper locality in 1.6.0-SNAPSHOT.  Before that change Accumulo would always store IP addrs.  Code like the following in o.a.a.c.client.mapreduce.AbstractInputFormat.getSplits() would translate IPs to hostnames.  
> {code:java}
>        if (location == null) {
>           InetAddress inetAddress = InetAddress.getByName(ip);
>           location = inetAddress.getHostName();
>           hostNameCache.put(ip, location);
>         }
> {code}
> In my case I configured Accumulo to use hostnames, but not fully qualified ones.  So I think the above code was just passing the non-quallified hostname through.   I suspected hadoop wanted FQDN and changed the code to the following and mappers ran locally.  I need to confirm what hadoop is expecting.  I think the above code will result in a FQDN if given an IP,  so this is not an issue for 1.4 or 1.5.
> {code:java}
>        if (location == null) {
>           InetAddress inetAddress = InetAddress.getByName(ip);
>           location = inetAddress.getCanonicalHostName();
>           hostNameCache.put(ip, location);
>         }
> {code}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)