[jira] [Commented] (TWILL-262) YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TWILL-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16634238#comment-16634238 ] 

ASF GitHub Bot commented on TWILL-262:
--------------------------------------

Github user chtyim commented on a diff in the pull request:

    https://github.com/apache/twill/pull/71#discussion_r221661663
  
    --- Diff: twill-yarn/src/main/java/org/apache/twill/internal/yarn/YarnUtils.java ---
    @@ -345,6 +388,28 @@ private static FileSystem getFileSystem(LocationFactory locationFactory) throws
         return null;
       }
     
    +  private static void handleLogAction(Exception e) {
    --- End diff --
    
    This method is unnecessary, better log it in place. E.g. in the static initializer above:
    
    ```java
    static {
      try {
        Class dfsUtilsClientClazz = Class.forName("org.apache.hadoop.hdfs.DFSUtilClient");
        getHaNnRpcAddressesMethod = dfsUtilsClientClazz.getMethod("getHaNnRpcAddresses",
            Configuration.class);
        hasDFSUtilClient = true;
      } catch (ClassNotFoundException e) {
        // Expected for Hadoop version < 2.8, hence log it as debug only to no polluting the logs
        LOG.debug("No DFSUtilClient found", e);
      } catch (NoSuchMethodException e) {
        // This is unexpected for not founding the getHaNnRpcAddresses method if the DFSUtilClient class exists
        LOG.warn("No DFSUtilClient.getHaNnRpcAddresses method found. Getting HA NameNode address might fail.", e);
      }
    }
    ```


> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8 
> ----------------------------------------------------------------------------------------------------------------
>
>                 Key: TWILL-262
>                 URL: https://issues.apache.org/jira/browse/TWILL-262
>             Project: Apache Twill
>          Issue Type: Bug
>          Components: yarn
>    Affects Versions: 0.8.0, 0.9.0, 0.11.0, 0.12.0, 0.13.0
>            Reporter: Hongyuan Li
>            Priority: Major
>         Attachments: errors.txt
>
>
> YarnUtils#cloneHaNnCredentials uses DFSUtil#getHaNnRpcAddresses, which is removed from DFSUtils from hadoop-2.8 
> current code
> {code}
>   public static void cloneHaNnCredentials(Configuration config) throws IOException {
>     ……
>     // Loop through all name services. Each name service could have multiple name node associated with it.
>     for (Map.Entry<String, Map<String, InetSocketAddress>> entry : DFSUtil.getHaNnRpcAddresses(config).entrySet()) {
>       String nsId = entry.getKey();
>       Map<String, InetSocketAddress> addressesInNN = entry.getValue();
>       if (!HAUtil.isHAEnabled(config, nsId) || addressesInNN == null || addressesInNN.isEmpty()) {
>         continue;
>       }
>  ……
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)