[jira] [Created] (AIRFLOW-622) airflow cannot recover itself after mysql connection was once gone away and not trigger an error from email alert

["SHENG JIANG TAO (JIRA)" <ji...@apache.org>]

SHENG JIANG TAO created AIRFLOW-622:
---------------------------------------

             Summary: airflow cannot recover itself after mysql connection was once gone away and not trigger an error from email alert
                 Key: AIRFLOW-622
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-622
             Project: Apache Airflow
          Issue Type: Bug
    Affects Versions: Airflow 1.6.2
         Environment: airflow 1.6.1
Ubuntu 14.04
            Reporter: SHENG JIANG TAO
            Priority: Critical


This issue occurs once the connections with airflow mysql gone away, even it recovered soon, but airflow hang there and print the following errors. 
Base on the suggestion of mysql, Session rollback method is needed.

Also not alert mail for this case, and no retry, just silently wrap the log files.

2016-11-10 16:04:30,361 [ERROR] file:jobs.py Line:582 - This Session's transaction has been rolled back by a nested rollback() call.  To begin a new transaction, issue Session.rollback() first.
Traceback (most recent call last):
  File "/datayes/mercury/anaconda/lib/python2.7/site-packages/airflow/jobs.py", line 580, in _execute
    self.prioritize_queued(executor=executor, dagbag=dagbag)
  File "/datayes/mercury/anaconda/lib/python2.7/site-packages/airflow/utils.py", line 122, in wrapper
    result = func(*args, **kwargs)
  File "/datayes/mercury/anaconda/lib/python2.7/site-packages/airflow/jobs.py", line 482, in prioritize_queued
    pools = {p.pool: p for p in session.query(models.Pool).all()}
  File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py", line 2584, in all
    return list(self)
  File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py", line 2732, in __iter__
    return self._execute_and_instances(context)
  File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py", line 2745, in _execute_and_instances
    close_with_result=True)
  File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py", line 2736, in _connection_from_session
    **kw)
  File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/session.py", line 905, in connection
    execution_options=execution_options)
  File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/session.py", line 910, in _connection_for_bind
    engine, execution_options)
  File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/session.py", line 313, in _connection_for_bind
    self._assert_active()
  File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/session.py", line 218, in _assert_active
    "This Session's transaction has been rolled back "
InvalidRequestError: This Session's transaction has been rolled back by a nested rollback() call.  To begin a new transaction, issue Session.rollback() first.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)