[jira] [Created] (DRILL-4591) Extend config system with site, node property files

["Paul Rogers (JIRA)" <ji...@apache.org>]

Paul Rogers created DRILL-4591:
----------------------------------

             Summary: Extend config system with site, node property files
                 Key: DRILL-4591
                 URL: https://issues.apache.org/jira/browse/DRILL-4591
             Project: Apache Drill
          Issue Type: Improvement
            Reporter: Paul Rogers
            Priority: Minor


Today Drill provides the drill-override.conf file to set Drill properties, and the drill-env.sh file to provide custom launch properties. Today, most users seem to have a copy of DRILL_HOME per node, and thus they copy these two files per-node.

The result is that the two files act as both the overall "site" configuration (for all nodes) and the "per-node" configuration for that one node.

The approach works, but is awkward. Ideally, provide the option to have two files: one for the site, one for the node.

drill-override.conf, site-env.sh: For the site
drill-node.conf, drill-env.sh: For the particular node

The improvement becomes more important if a user employs NFS, MapR FS or YARN to automatically deploy the site-wide files. In that case, the site files cannot also act as per-node files.

Note that, above, site-env.sh is new. The existing file is supposed to be per-node, so the new file is site-wide.

By contrast, the drill-node.conf per-node configuration is new as the existing file is supposed to be site-wide.

The site-wide site-env.sh might, say, point to custom locations for log, conf, pid or other file locations, or set custom JVM options.

Similarly, the per-node drill-node.conf might customize the per-node memory or other resources if one node has a different amount of physical memory or other resources.

The existing files are identified with the DRILL_CONF_DIR environment variable. Add a new DRILL_SITE_CONF_DIR variable to point to the site-wide files.

When loading, per-node files are loaded after the site-wide files (so that per-node values override site-wide values.)

As described, the change should be fully backward compatible. The change will also work with a YARN solution that "localizes" (copies to the local node) the site-wide files. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)