[jira] [Created] (DRILL-6521) When using Hadoop configs, allow dfs connection to be unset

["Paul Rogers (JIRA)" <ji...@apache.org>]

Paul Rogers created DRILL-6521:
----------------------------------

             Summary: When using Hadoop configs, allow dfs connection to be unset
                 Key: DRILL-6521
                 URL: https://issues.apache.org/jira/browse/DRILL-6521
             Project: Apache Drill
          Issue Type: Improvement
    Affects Versions: 1.13.0
            Reporter: Paul Rogers


Drill usually works stand-alone. In a Hadoop HDFS environment, the docs. say to set the HDFS configuration in the storage plugin config.

In a MapR installation, MaprFS settings are applied automatically.

However, when Drill runs on an existing HDFS cluster, one must often provide more than the simple HDFS URL. Particularly in a secure cluster, other configuration settings are also needed. At present, these must be copied out of the HDFS config files into the Drill storage plugin config, and the two must be updated in tandem. Clearly less than ideal.

Drill does allow the user to add Hadoop configs to the class path. (Though, it looks like in recent releases the previous {{HADOOP_CONF}} setting has been removed.) The user can edit {{drill-env.sh}} to add the Hadoop class path to {{EXTN_CLASSPATH}}. (But see DRILL-6520.)

This is all good, but Drill still requires that the "dfs" storage plugin config contain a connection. Omit the connection and we get:
{noformat}
Please retry: Error while creating/ updating storage : The value of property fs.defaultFS must not be null
{noformat}
Would expect to be able to omit this property if the value is provided by the standard Hadoop {{core-site.xml}} file (and to do so without Drill crashing, per DRILL-6520).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)