[jira] [Commented] (DRILL-5547) Drill config options and session options do not work as intended

["Paul Rogers (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DRILL-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16045021#comment-16045021 ] 

Paul Rogers commented on DRILL-5547:
------------------------------------

Proposal: Link the two config systems to allow custom distributions, or special cases, to set the default values for system/session options using the config system.

At present, the runtime system/session option system is distinct from the boot-time config system. The defaults for system/session options are hard coded in {{ExecConstants.java}} (among other places). The way to change system/session globally is to issue an ALTER SYSTEM command.

It also turns out that, in a number of areas, we have ad-hoc linkage between the two systems. For example, for the code gen compiler, we have logic that (tries, but fails) to check if the system option is set, and if not, use a config option.

Suppose a custom software distribution of Drill wants to ship with system option defaults other than what Apache Drill defines. The only ways to do that today are:

# A custom build of Drill with new hard-coded defaults.
# The client sets the options on each connection via ALTER SESSION.
# The installer fires up Drill and issues an ALTER SYSTEM command.

Option 1 works, but it requires  a custom build and cherry-picked changes going forward. Works, but less than ideal.

For option 2, the client of the distribution (potentially many!) would have to alter the session options in a way that that the distribution needs. This is not terribly reliable.

Option 3 requires the distribution installer to start Drill, issue ALTER SYSTEM options, and shut down Drill. It works, but is awkward.

Let’s go back to how system options work. They are in three layers:

* Session options
* System options
* Hard-coded defaults in {{ExecConstants.java}} (and other places)

So, as we said, one way to create query service is to alter the code in {{ExecConstants}} to have new defaults. 

Maybe we can do what UI tools often do for strings: externalize them in a Java properties file. They are still defaults defined by Drill (not the user), but they are defined in a properties file rather than in Java source code. Distributions of Drill are free to alter the defaults (just as localizations alter user-visible strings in a UI tool.)

But, wait, Drill already has such a system. Instead of using properties files, why not use the HOCON TypeSafe system we already have? Another JIRA is introducing a new {{drill-distrib.conf}} file that sits between the existing Drill-provide {{drill-module.conf}} and the user-provided {{drill-override.conf}} file. This file is parallel to the {{distrib-env.sh}} file introduced in 1.8.

The key is to realize is that *users* won’t alter these defaults. Only the installer for each custom Drill distribution, via {{drill-distrib.conf}}.

Could a use change the options in their {{drill-module.conf}} file, change it again via ALTER SYSTEM, and get confused? Probably. But the solution is easy: don't do that.

The alternative is to leave the defaults hard-coded (with the issues noted above), or externalize them into Java properties files, then implement parsers for the values (which, as it turns out, we already have in the form of validators.)

Thoughts? Comments? Suggestions?

> Drill config options and session options do not work as intended
> ----------------------------------------------------------------
>
>                 Key: DRILL-5547
>                 URL: https://issues.apache.org/jira/browse/DRILL-5547
>             Project: Apache Drill
>          Issue Type: Bug
>          Components:  Server
>    Affects Versions: 1.10.0
>            Reporter: Karthikeyan Manivannan
>            Assignee: Venkata Jyothsna Donapati
>             Fix For: Future
>
>
> In Drill, session options should take precedence over config options. But several of these session options are assigned hard-coded default values when the option validators are initialized. Because of this config options will never be read and honored even if the user did not specify the session option. 
> ClassCompilerSelector.JAVA_COMPILER_VALIDATOR uses CompilerPolicy.DEFAULT as the default value. This default value gets into the session options map via the initialization of validators in SystemOptionManager. 
> Now any piece of code that tries to check if a session option is set will never see a null, so it will always use that value and never try to look into the config options. For example, in the following piece of code from ClassCompilerSelector (), the policy will never be read from the config file.
> policy = CompilerPolicy.valueOf((value != null) ? value.string_val.toUpperCase() : config.getString(JAVA_COMPILER_CONFIG).toUpperCase());



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)