[jira] [Created] (HADOOP-12512) hadoop fs -ls / fails with below error when we use Custom -Dhadoop.root.logger

["Prabhu Joseph (JIRA)" <ji...@apache.org>]

Prabhu Joseph created HADOOP-12512:
--------------------------------------

             Summary: hadoop fs -ls / fails with below error when we use Custom -Dhadoop.root.logger 
                 Key: HADOOP-12512
                 URL: https://issues.apache.org/jira/browse/HADOOP-12512
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs
    Affects Versions: 2.7.0
            Reporter: Prabhu Joseph



hadoop fs -ls / fails with below error when we use Custom -Dhadoop.root.logger and creates Configuration object and adds defaultResource custom-conf.xml with quiet = false.
custom-conf.xml is optional configuration.

Exception in thread "main" java.lang.RuntimeException: custom-conf.xml not found
        at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2612)
        at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2531)
        at
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2444)
        at org.apache.hadoop.conf.Configuration.set(Configuration.java:1156)
        at org.apache.hadoop.conf.Configuration.set(Configuration.java:1128)
        at
org.apache.hadoop.conf.Configuration.setBoolean(Configuration.java:1464)
        at
org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:321)
        at
org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:487)
        at
org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:170)
        at
org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:153)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:64)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
        at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)
		
		

ISSUE:
######

There is a Logic issue in Configuration class and defaultResources list.

Configuration is shared by classes. Configuration has a shared list of default resources added by the classes.

If class A wants x,y,z resources and says all are optional using quiet = false, Configuration loads if they are present else skips and adds them to list.

Now shared list i.e defaultResources has x,y,z

Now if class B wants x resource and says it as mandatory , loadResources scans the entire list and treats them mandatory. So during scan of y, it will fail.

where A is Custom Class
and B is FsShell

FsShell checks for custom-conf.xml and treats it mandatory and fails.

1. The mandatory/optional has to be resource wise. [OR]
2. defaultResources should not be shared. 

Both of them looks complex. And simple fix is the below.

1. when loadResource skips initially if resource not found, it has to remove the entry from defaultResource list as well. There is no use in having a resource 
which is not at classpath in the list.


CODE CHANGE:  class org.apache.hadoop.conf.Configuration
############

private Resource loadResource(Properties properties, Resource wrapper, boolean
quiet) {}
 ......

 if (root == null) {
        if (doc == null) {
                  if (quiet) {
        defaultResources.remove(resource);  // FIX: During skip, remove
Resource from shared list 
                    return null;
                  }
        throw new RuntimeException(resource + " not found");
        }
     root = doc.getDocumentElement();
 }

 ........


Tested after code fix, rans successfully.







--
This message was sent by Atlassian JIRA
(v6.3.4#6332)