[jira] [Commented] (HADOOP-12512) hadoop fs -ls / fails when we use Custom -Dhadoop.root.logger

["Daniel Templeton (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-12512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14974609#comment-14974609 ] 

Daniel Templeton commented on HADOOP-12512:
-------------------------------------------

[~Prabhu Joseph], since you've already done the work, could you post a patch?

Checkout a clean version of trunk, make your change, then run:

{noformat}
git diff trunk > HADOOP-12512.001.patch
{format}

Add the patch as a file to the JIRA (under the More pull-down) and then click the Patch Submitted button.

It might not be a bad idea to first run:

{noformat}
git diff trunk --check
{noformat}

just to make sure Jenkins won't complain about whitespace.  Let me know if you need any help.

> hadoop fs -ls / fails when we use Custom -Dhadoop.root.logger 
> --------------------------------------------------------------
>
>                 Key: HADOOP-12512
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12512
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.7.0
>            Reporter: Prabhu Joseph
>
> hadoop fs -ls / fails with below error when we use Custom -Dhadoop.root.logger that creates Configuration object and adds defaultResource custom-conf.xml with quiet = false.
> custom-conf.xml is optional configuration.
> Exception in thread "main" java.lang.RuntimeException: custom-conf.xml not found
>         at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2612)
>         at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2531)
>         at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2444)
>         at org.apache.hadoop.conf.Configuration.set(Configuration.java:1156)
>         at org.apache.hadoop.conf.Configuration.set(Configuration.java:1128)
>         at
> org.apache.hadoop.conf.Configuration.setBoolean(Configuration.java:1464)
>         at
> org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:321)
>         at
> org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:487)
>         at
> org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:170)
>         at
> org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:153)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:64)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>         at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)
> 		
> 		
> ISSUE:
> ######
> There is a Logic issue in Configuration class and defaultResources list.
> Configuration is shared by classes. Configuration has a shared list of default resources added by the classes.
> If class A wants x,y,z resources and says all are optional using quiet = false, Configuration loads if they are present else skips and adds them to list.
> Now shared list i.e defaultResources has x,y,z
> Now if class B wants x resource and says it as mandatory , loadResources scans the entire list and treats them mandatory. So during scan of y, it will fail.
> where A is Custom Class
> and B is FsShell
> FsShell checks for custom-conf.xml and treats it mandatory and fails.
> 1. The mandatory/optional has to be resource wise. [OR]
> 2. defaultResources should not be shared. 
> Both of them looks complex. And simple fix is the below.
> 1. when loadResource skips initially if resource not found, it has to remove the entry from defaultResource list as well. There is no use in having a resource 
> which is not at classpath in the list.
> CODE CHANGE:  class org.apache.hadoop.conf.Configuration
> ############
> private Resource loadResource(Properties properties, Resource wrapper, boolean
> quiet) {}
>  ......
>  if (root == null) {
>         if (doc == null) {
>                   if (quiet) {
>         defaultResources.remove(resource);  // FIX: During skip, remove
> Resource from shared list 
>                     return null;
>                   }
>         throw new RuntimeException(resource + " not found");
>         }
>      root = doc.getDocumentElement();
>  }
>  ........
> Tested after code fix, runs successfully.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)