You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@drill.apache.org by "Paul Rogers (JIRA)" <ji...@apache.org> on 2016/05/20 05:46:12 UTC
[jira] [Comment Edited] (DRILL-4591) Extend config system with
distrib, site, node property files
[ https://issues.apache.org/jira/browse/DRILL-4591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292759#comment-15292759 ]
Paul Rogers edited comment on DRILL-4591 at 5/20/16 5:46 AM:
-------------------------------------------------------------
A simpler, but still useful, solution is to add three minor revisions, all backward compatible.
1. Split drill-env.sh into three files. Drill defaults move into drill-config.sh. Distribution-specific settings go into a new (optional) distrib-env.sh file. User-specified settings stay in drill-env.sh (which now holds ONLY user entries.)
2. Check for a jars directory within $DRILL_CONFIG_DIR. If present, add it to the class path.
4. If $DRILL_CONFIG_DIR points to a location other than $DRILL_HOME/conf, then add $DRILL_HOME/conf to the class path after $DRILL_CONFIG_DIR. That way, Drill will pick up the default logback.xml file if the user does not provide a custom version.
The above allows the user to set up a Drill site directory as follows:
my-drill
. drill-override.conf
. drill-env.sh
. jars
. . myCustom.jar
Launch Drill with:
drillbit.sh --config /path/to/my-drill
Or
export DRILL_CONF_DIR=/path/to/my-drill
drillbit.sh
The result is that all user files go into the custom config directory, no user entries or files go anywhere in $DRILL_HOME. Upgrading is now trivial. When Drill is run under YARN, only the config files need be uploaded to DFS for a config change rather than the entire Drill distribution, resulting in faster start of a YARN-managed Drill cluster.
The one item that this abbreviated proposal does not address is node-specific configuration. However, that seems a rare case. Users that use node-specific settings probably need to change only drill-env.sh, which can be done by sourcing a node-specific script inside drill-env.sh.
was (Author: paul-rogers):
A simpler, but still useful, solution is to add three minor revisions, all backward compatible.
1. Split drill-env.sh into three files. Drill defaults move into drill-config.sh. Distribution-specific settings go into a new (optional) distrib-env.sh file. User-specified settings stay in drill-env.sh (which now holds ONLY user entries.)
2. Check for a jars directory within $DRILL_CONFIG_DIR. If present, add it to the class path.
4. If $DRILL_CONFIG_DIR points to a location other than $DRILL_HOME/conf, then add $DRILL_HOME/conf to the class path after $DRILL_CONFIG_DIR. That way, Drill will pick up the default logback.xml file if the user does not provide a custom version.
The above allows the user to set up a Drill site directory as follows:
my-drill
|- drill-override.conf
|- drill-env.sh
|- jars
|- myCustom.jar
Launch Drill with:
drillbit.sh --config /path/to/my-drill
Or
export DRILL_CONF_DIR=/path/to/my-drill
drillbit.sh
The result is that all user files go into the custom config directory, no user entries or files go anywhere in $DRILL_HOME. Upgrading is now trivial. When Drill is run under YARN, only the config files need be uploaded to DFS for a config change rather than the entire Drill distribution, resulting in faster start of a YARN-managed Drill cluster.
The one item that this abbreviated proposal does not address is node-specific configuration. However, that seems a rare case. Users that use node-specific settings probably need to change only drill-env.sh, which can be done by sourcing a node-specific script inside drill-env.sh.
> Extend config system with distrib, site, node property files
> ------------------------------------------------------------
>
> Key: DRILL-4591
> URL: https://issues.apache.org/jira/browse/DRILL-4591
> Project: Apache Drill
> Issue Type: Improvement
> Reporter: Paul Rogers
> Attachments: Drill-on-YARNDirectoryStructures.pdf
>
>
> Today Drill provides the drill-override.conf file to set Drill properties, and the drill-env.sh file to provide custom launch properties. Today, most users seem to have a copy of DRILL_HOME per node, and thus they copy these two files per-node. The result is that the two files act as both the overall "site" configuration (for all nodes) and the "per-node" configuration for that one node.
> In addition, some distributions of Drill (such as MapR), modify the "user" config files with settings for that distribution. Now, the same files hold settings for the distribution, site and node.
> The approach works, but is awkward. Ideally, provide the option to have three sets of files: for the distribution, site, and node.
> The proposal is to extend configuration to provide additional levels:
> * Drill defaults (drill default and module conf files, code in drill-config.sh)
> * Distribution settings (special JVM settings, say)
> * Site settings (standard log or spill file locations)
> * Node settings
> * Launch settings (environment variables, -Dname=value options)
> The improvement becomes more important if a user employs NFS, MapR FS or YARN to automatically deploy the site-wide files. In that case, the site files cannot also act as per-node files.
> The improvement also simplifies upgrades. Today, users must copy customizations from and old to a new install. With the revision, Drill files are complely separated from user files, making upgrades (of software) easier.
> For backward compatibility, the site and node directories are optional and ignored if the environment variables are not set. The site and node config files should be optional: skip them if they do not exist.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)