[jira] [Closed] (AIRFLOW-622) airflow cannot recover itself after mysql connection was once gone away and not trigger an error from email alert

["Daniel Imberman (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daniel Imberman closed AIRFLOW-622.
-----------------------------------
    Resolution: Auto Closed

> airflow cannot recover itself after mysql connection was once gone away and not trigger an error from email alert
> -----------------------------------------------------------------------------------------------------------------
>
>                 Key: AIRFLOW-622
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-622
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: database
>    Affects Versions: 1.6.2
>         Environment: airflow 1.6.1
> Ubuntu 14.04
>            Reporter: SHENG JIANG TAO
>            Priority: Critical
>              Labels: mysql
>
> This issue occurs once the connections with airflow mysql gone away, even it recovered soon, but airflow hang there and print the following errors. 
> Base on the suggestion of mysql, Session rollback method is needed.
> Also not alert mail for this case, and no retry, just silently wrap the log files.
> 2016-11-10 16:04:30,361 [ERROR] file:jobs.py Line:582 - This Session's transaction has been rolled back by a nested rollback() call.  To begin a new transaction, issue Session.rollback() first.
> Traceback (most recent call last):
>   File "/datayes/mercury/anaconda/lib/python2.7/site-packages/airflow/jobs.py", line 580, in _execute
>     self.prioritize_queued(executor=executor, dagbag=dagbag)
>   File "/datayes/mercury/anaconda/lib/python2.7/site-packages/airflow/utils.py", line 122, in wrapper
>     result = func(*args, **kwargs)
>   File "/datayes/mercury/anaconda/lib/python2.7/site-packages/airflow/jobs.py", line 482, in prioritize_queued
>     pools = {p.pool: p for p in session.query(models.Pool).all()}
>   File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py", line 2584, in all
>     return list(self)
>   File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py", line 2732, in __iter__
>     return self._execute_and_instances(context)
>   File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py", line 2745, in _execute_and_instances
>     close_with_result=True)
>   File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/query.py", line 2736, in _connection_from_session
>     **kw)
>   File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/session.py", line 905, in connection
>     execution_options=execution_options)
>   File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/session.py", line 910, in _connection_for_bind
>     engine, execution_options)
>   File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/session.py", line 313, in _connection_for_bind
>     self._assert_active()
>   File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/session.py", line 218, in _assert_active
>     "This Session's transaction has been rolled back "
> InvalidRequestError: This Session's transaction has been rolled back by a nested rollback() call.  To begin a new transaction, issue Session.rollback() first.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)