[jira] [Created] (AIRFLOW-328) Refactor configuration.py to avoid seperate default dict specification

["Jeremiah Lowin (JIRA)" <ji...@apache.org>]

Jeremiah Lowin created AIRFLOW-328:
--------------------------------------

             Summary: Refactor configuration.py to avoid seperate default dict specification
                 Key: AIRFLOW-328
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-328
             Project: Apache Airflow
          Issue Type: Improvement
    Affects Versions: Airflow 1.7.1.3
            Reporter: Jeremiah Lowin
            Assignee: Jeremiah Lowin
            Priority: Minor


Currently, configuration has a number of moving parts:
1. a string template for airflow.cfg
2. the actual airflow.cfg file in airflow home
3. a dict of config defaults
4. a configparser subclass that first accesses env vars, then bash commands, then airflow.cfg, then the defaults dict

Maintaining both the string template (#1) and the defaults dict (#3) is redundant and leads to errors if users don't know that *both* need to be updated in order to create new configuration options. Moreover, it necessitates the use of a configparser subclass to read the defaults (the subclass is still required for env vars/bash commands but setting that aside for a moment...).

I think the issue of re-specificying defaults in both template and dict can be handled with a simple refactor:
1. have the configparser load the string template
2. have the configparser load the actual airflow.cfg
This way, any options specified in airflow.cfg will overwrite the template value, but any missing options will fall back on the template. The defaults dict is no longer needed.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)