[jira] [Created] (AIRFLOW-4527) Execution time out doesn't work after MySql Lost connection while refreshing from db

["Bharath Palaksha (JIRA)" <ji...@apache.org>]

Bharath Palaksha created AIRFLOW-4527:
-----------------------------------------

             Summary: Execution time out doesn't work after MySql Lost connection while refreshing from db
                 Key: AIRFLOW-4527
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-4527
             Project: Apache Airflow
          Issue Type: Bug
            Reporter: Bharath Palaksha


{{I have setup airflow with mysql as metastore. When there is a network issue and task fails with an exception, airflow tries to refresh status from db and gets a connection error - This results in task getting stuck in running and execution time out doesn't apply for this. }}

{{There is no retry for mysql connection error and it never fails the task.}}

 

 Scheduler heartbeat got an exception: (MySQLdb._exceptions.OperationalError) (2013, "Lost connection to MySQL server at 'reading authorization packet', system error: 104") (Background on this error at: http://sqlalche.me/e/e3q8)
{base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data Traceback (most recent call last):
{base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data File "/usr/local/bin/airflow", line 32, in <module>
{base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data args.func(args)
{base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data File "/usr/local/lib/python2.7/site-packages/airflow/utils/cli.py", line 74, in wrapper
{base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data return f(*args, **kwargs)
{base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data File "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 526, in run
{base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data _run(args, dag, ti)
{base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data File "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 445, in _run
{base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data pool=args.pool,
{base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data File "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 73, in wrapper
{base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data return func(*args, **kwargs)
{base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1692, in _run_raw_task
{base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data self.refresh_from_db()
{base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data File "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 73, in wrapper
{base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data return func(*args, **kwargs)
{base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1218, in refresh_from_db
{base_task_runner.py:101} INFO - Job 989226: Subtask count_cust_shipped_data ti = qry.first()

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)