[jira] [Updated] (DRILL-4591) Extend config system with site, node property files

["Paul Rogers (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/DRILL-4591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Paul Rogers updated DRILL-4591:
-------------------------------
    Description: 
Today Drill provides the drill-override.conf file to set Drill properties, and the drill-env.sh file to provide custom launch properties. Today, most users seem to have a copy of DRILL_HOME per node, and thus they copy these two files per-node.

The result is that the two files act as both the overall "site" configuration (for all nodes) and the "per-node" configuration for that one node.

The approach works, but is awkward. Ideally, provide the option to have two files: one for the site, one for the node.

drill-override.conf, drill-env.sh: For the site
drill-override.conf, drill-env.sh: For the particular node

The site-wide site-env.sh might, say, point to custom locations for log, conf, pid or other file locations, or set custom JVM options.

Similarly, the per-node drill-node.conf might customize the per-node memory or other resources if one node has a different amount of physical memory or other resources.

The improvement becomes more important if a user employs NFS, MapR FS or YARN to automatically deploy the site-wide files. In that case, the site files cannot also act as per-node files.

Existing installs have only one each of drill-override.conf, drill-env.sh, copied to each node and possibly modified. For compatibility, the existing files become the site-wide files. When converting to a split-setup, copy the site-wide files as before. But, rather than modifying the site-wide files per-node, create a new per-load location, create new per-node files, and place the per-node modifications in those files. The per-node files need not change on following upgrades, just copy the site-wide files and that's it.

We assume that DRILL_CONF_DIR points to the site-wide files. It defaults to $DRILL_HOME/conf or in /etc/drill/conf. To use node-specific files, the admin must set up a new, per-node location and identify that location using a proposed DRILL_NODE_CONF_DIR variable.

The config files should be optional: skip them if they do not exist (or, for node files, skip them if DRILL_NODE_CONF_DIR is not set.)

When loading, per-node files are loaded after the site-wide files (so that per-node values override site-wide values.)

The change should be fully backward compatible. The change will also work with a YARN solution that "localizes" (copies to the local node) the site-wide files. 

  was:
Today Drill provides the drill-override.conf file to set Drill properties, and the drill-env.sh file to provide custom launch properties. Today, most users seem to have a copy of DRILL_HOME per node, and thus they copy these two files per-node.

The result is that the two files act as both the overall "site" configuration (for all nodes) and the "per-node" configuration for that one node.

The approach works, but is awkward. Ideally, provide the option to have two files: one for the site, one for the node.

drill-override.conf, site-env.sh: For the site
drill-node.conf, drill-env.sh: For the particular node

The improvement becomes more important if a user employs NFS, MapR FS or YARN to automatically deploy the site-wide files. In that case, the site files cannot also act as per-node files.

Note that, above, site-env.sh is new. The existing file is supposed to be per-node, so the new file is site-wide.

By contrast, the drill-node.conf per-node configuration is new as the existing file is supposed to be site-wide.

The site-wide site-env.sh might, say, point to custom locations for log, conf, pid or other file locations, or set custom JVM options.

Similarly, the per-node drill-node.conf might customize the per-node memory or other resources if one node has a different amount of physical memory or other resources.

The existing files are identified with the DRILL_CONF_DIR environment variable. Add a new DRILL_SITE_CONF_DIR variable to point to the site-wide files.

When loading, per-node files are loaded after the site-wide files (so that per-node values override site-wide values.)

As described, the change should be fully backward compatible. The change will also work with a YARN solution that "localizes" (copies to the local node) the site-wide files. 


> Extend config system with site, node property files
> ---------------------------------------------------
>
>                 Key: DRILL-4591
>                 URL: https://issues.apache.org/jira/browse/DRILL-4591
>             Project: Apache Drill
>          Issue Type: Improvement
>            Reporter: Paul Rogers
>            Priority: Minor
>
> Today Drill provides the drill-override.conf file to set Drill properties, and the drill-env.sh file to provide custom launch properties. Today, most users seem to have a copy of DRILL_HOME per node, and thus they copy these two files per-node.
> The result is that the two files act as both the overall "site" configuration (for all nodes) and the "per-node" configuration for that one node.
> The approach works, but is awkward. Ideally, provide the option to have two files: one for the site, one for the node.
> drill-override.conf, drill-env.sh: For the site
> drill-override.conf, drill-env.sh: For the particular node
> The site-wide site-env.sh might, say, point to custom locations for log, conf, pid or other file locations, or set custom JVM options.
> Similarly, the per-node drill-node.conf might customize the per-node memory or other resources if one node has a different amount of physical memory or other resources.
> The improvement becomes more important if a user employs NFS, MapR FS or YARN to automatically deploy the site-wide files. In that case, the site files cannot also act as per-node files.
> Existing installs have only one each of drill-override.conf, drill-env.sh, copied to each node and possibly modified. For compatibility, the existing files become the site-wide files. When converting to a split-setup, copy the site-wide files as before. But, rather than modifying the site-wide files per-node, create a new per-load location, create new per-node files, and place the per-node modifications in those files. The per-node files need not change on following upgrades, just copy the site-wide files and that's it.
> We assume that DRILL_CONF_DIR points to the site-wide files. It defaults to $DRILL_HOME/conf or in /etc/drill/conf. To use node-specific files, the admin must set up a new, per-node location and identify that location using a proposed DRILL_NODE_CONF_DIR variable.
> The config files should be optional: skip them if they do not exist (or, for node files, skip them if DRILL_NODE_CONF_DIR is not set.)
> When loading, per-node files are loaded after the site-wide files (so that per-node values override site-wide values.)
> The change should be fully backward compatible. The change will also work with a YARN solution that "localizes" (copies to the local node) the site-wide files. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)