[jira] [Created] (AIRFLOW-3973) `airflow initdb` logs errors when `Variable` is used in DAGs and Postgres is used for the internal database

["Elliott Shugerman (JIRA)" <ji...@apache.org>]

Elliott Shugerman created AIRFLOW-3973:
------------------------------------------

             Summary: `airflow initdb` logs errors when `Variable` is used in DAGs and Postgres is used for the internal database
                 Key: AIRFLOW-3973
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3973
             Project: Apache Airflow
          Issue Type: Bug
            Reporter: Elliott Shugerman
            Assignee: Elliott Shugerman


h2. Example

{{ERROR [airflow.models.DagBag] Failed to import: /home/elliott/clean-airflow/dags/dag.py Traceback (most recent call last): File "/home/elliott/.virtualenvs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1236, in _execute_context cursor, statement, parameters, context File "/home/elliott/.virtualenvs/airflow/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 536, in do_execute cursor.execute(statement, parameters) psycopg2.ProgrammingError: relation "variable" does not exist LINE 2: FROM variable}}
h2. Explanation

The first thing {{airflow initdb}} does is run the Alembic migrations. All migrations are run in one transaction. Most tables, including the {{variable}} table, are defined in the initial migration. A [later migration|https://github.com/apache/airflow/blob/master/airflow/migrations/versions/cc1e65623dc7_add_max_tries_column_to_task_instance.py] imports and initializes {{models.DagBag}}. Upon initialization, {{DagBag}} calls its {{collect_dags}} method, which scans the DAGs directory and attempts to load all DAGs it finds. When it loads a DAG that uses a {{Variable}}, it will query the database to see if that {{Variable}} is defined in the {{variable}} table. It's not clear to me how exactly the connection for that query is created, but I think it is a fair assumption that it does _not_ use the same transaction that is used to run the migrations. Since the migrations are not yet complete, and all migrations are run in one transaction, the migration that creates the {{variable}} table has not yet been committed, and therefore the table does not exist to any other connection/transaction. This raises {{ProgrammingError}}, which is caught and logged by {{collect_dags}}.

NOTE: This does not occur with the default SQLite database.
h2. Proposed Solution

Run each Alembic migration in its own transaction. I will be opening a pull request which accomplishes this shortly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)