[jira] [Commented] (DRILL-5547) Drill config options and session options do not work as intended

["Paul Rogers (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DRILL-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16027074#comment-16027074 ] 

Paul Rogers commented on DRILL-5547:
------------------------------------

This is quite a complicated issue!

Drill has two different "config" systems which operate as two independent name spaces. Each works well by itself; the problem is where the systems interact.

First, we have the TypeSafe, HOCON-based boot-time config system. This system is a stack of layers where upper levels override lower levels:

* Java sytem properties ({{-Dname=value}} command line options).
* New {{distrib.conf}} file for distribution-specific options. (Work in progress at the time of this note.)
* {{drill-override.conf}} file in {{$DRILL_SITE}} (which defaults to {{$DRILL_HOME/conf}}).
* The various module-specific {{drill-module.conf}} files.

These options are for options set *before* Drill starts.

The other system is the system/session option system. Values are set with:

{code}
ALTER SYSTEM SET `name`=value
ALTER SESSION SET `name`=value
{code}

The system/session system also provides an inheritance-style system:

* Session options take precedence
* Otherwise, the system option value is used
* Otherwise, a hard-coded default value is used

Here is where things start to get obscure. System/session options have a name and a _validator_. The validator checks that the string value matches the expected format (such as number, enum, etc.) The validator also provides a default value (set in code, typically in {{ExecConstants}}.) Until recently, an unset system/session option was null; the default was applied at look-up time by first looking up a value, checking if it is null, then using the validator's default.

But, to populate default values in the UI, a new table was created in {{SystemOptionManager}} that registers all the validators so that system/session options always have a default. This works very well, except for the case that this bug describes.

Code used to use the following protocol to connect the system/session and config option systems:

* Look up the system/session option. If found, use it because this means the admin or user set it.
* If not set, don't use the default from the validator.
* Instead, use a different key to look up a value in the config option system and use that as the setting.

Once the defaults were registered, the first lookup never returns null, and the config options are never set. One improvement had the net effect of breaking other features, unfortunately.

The goal here is to keep the ability to display defaults, but also to get defaults from the boot-time config system.

Essentially, we want to change the system/session options to:

* Session options take precedence
* Otherwise, the system option value is used
* Otherwise, map the session option to a config key and look up the default in the config system.

The key mapping is needed because the config system tends to use keys of the form "drill.exec.foo.bar", while the system/session options tend to be "foo.bar" or "exec.foo.bar".

One proposal is to declare a fixed config prefix, say "drill.exec.options" after which we use the session option name. So, session option "foo.bar" maps to "drill.exec.options.foo.bar" in the config system.

> Drill config options and session options do not work as intended
> ----------------------------------------------------------------
>
>                 Key: DRILL-5547
>                 URL: https://issues.apache.org/jira/browse/DRILL-5547
>             Project: Apache Drill
>          Issue Type: Bug
>          Components:  Server
>    Affects Versions: 1.10.0
>            Reporter: Karthikeyan Manivannan
>            Assignee: Venkata Jyothsna Donapati
>             Fix For: Future
>
>
> In Drill, session options should take precedence over config options. But several of these session options are assigned hard-coded default values when the option validators are initialized. Because of this config options will never be read and honored even if the user did not specify the session option. 
> ClassCompilerSelector.JAVA_COMPILER_VALIDATOR uses CompilerPolicy.DEFAULT as the default value. This default value gets into the session options map via the initialization of validators in SystemOptionManager. 
> Now any piece of code that tries to check if a session option is set will never see a null, so it will always use that value and never try to look into the config options. For example, in the following piece of code from ClassCompilerSelector (), the policy will never be read from the config file.
> policy = CompilerPolicy.valueOf((value != null) ? value.string_val.toUpperCase() : config.getString(JAVA_COMPILER_CONFIG).toUpperCase());



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)