[jira] [Created] (AIRFLOW-1642) An Alembic script not using scoped session causing deadlock

["Joy Gao (JIRA)" <ji...@apache.org>]

Joy Gao created AIRFLOW-1642:
--------------------------------

             Summary: An Alembic script not using scoped session causing deadlock
                 Key: AIRFLOW-1642
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1642
             Project: Apache Airflow
          Issue Type: Bug
            Reporter: Joy Gao
            Priority: Minor


The bug I'm about to describe is a more of an obscure edge case, however I think it's something still worth fixing.

After upgrading to airflow 1.9, while running `airflow resetdb` on my local machine (with mysql), I encountered a deadlock on the final alembic revision _d2ae31099d61 Increase text size for MySQL (not relevant for other DBs' text types)_.

The deadlock turned out to be caused by another earlier session that was created and left open in revision _cc1e65623dc7 add max tries column to task instance_. Notably the code below:

{code}
sessionmaker = sa.orm.sessionmaker()
session = sessionmaker(bind=connection)
dagbag = DagBag(settings.DAGS_FOLDER)
{code}

The session created here was not a `scoped_session`, so when the DAGs were being parsed in line 3 above, one of the DAG files makes a direct call to the class method `Variable.get()` to acquire an env variable, which makes a db query to the `variable` table, but raised a KeyError as the env variable was non-existent, thus holding the lock to the `variable` table as a result of that exception.

Later on, the latter alembic script `_cc1e65623dc7` needs to alter the `Variable` table. Instead of creating its own Session object, it attempts to reuse the same one as above. And because of the exception, it waits indefinitely to acquire the lock on that table. 

So the DAG file itself could have avoided the KeyError by providing a default value when calling Variable.get(). However I think it would be a good idea to avoid using unscoped sessions in general, as an exception could potentially occur in the future elsewhere.  The easiest fix is replacing *session = sessionmaker(bind=connection)* with *session = settings.Session()*, which is scoped. However, making a change on a migration script is going to make folks anxious.

If anyone have any thoughts on this, let me know! Thanks :)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)