You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Aizhamal Nurmamat kyzy (JIRA)" <ji...@apache.org> on 2019/05/17 20:03:04 UTC
[jira] [Updated] (AIRFLOW-3131) Start Date is not honoured in case
of task is failed due to Mysql connection failure
[ https://issues.apache.org/jira/browse/AIRFLOW-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aizhamal Nurmamat kyzy updated AIRFLOW-3131:
--------------------------------------------
Component/s: (was: db)
(was: cli)
database
> Start Date is not honoured in case of task is failed due to Mysql connection failure
> ------------------------------------------------------------------------------------
>
> Key: AIRFLOW-3131
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3131
> Project: Apache Airflow
> Issue Type: Bug
> Components: database
> Affects Versions: 1.9.0
> Reporter: Tanuj Gupta
> Priority: Minor
>
> Sometimes, when we trigger some DAG then task's start sate is seen as null in database. In this we have observed that "airflow run" command fails due to mysql connection failure and in turn task state is set as FAILED but start date is not updated in the database.
>
> Traceback (most recent call last): File "/usr/src/venv/bin/airflow", line 27, in <module> args.func(args) File "/usr/src/venv/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 387, in run run_job.run() File "/usr/src/venv/local/lib/python2.7/site-packages/airflow/jobs.py", line 193, in run id_ = self.id File "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 242, in __get__ return self.impl.get(instance_state(instance), dict_) File "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 594, in get value = state._load_expired(state, passive) File "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/state.py", line 608, in _load_expired self.manager.deferred_scalar_loader(self, toload) File "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 876, in load_scalar_attributes only_load_props=attribute_names) File "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 188, in load_on_ident identity_token=identity_token File "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 250, in load_on_pk_identity return q.one() File "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2947, in one ret = self.one_or_none() File "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2917, in one_or_none ret = list(self) File "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2988, in __iter__ return self._execute_and_instances(context) File "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 3009, in _execute_and_instances close_with_result=True) File "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 3018, in _get_bind_args **kw File "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 3000, in _connection_from_session conn = self.session.connection(**kw) File "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection execution_options=execution_options) File "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind engine, execution_options) File "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind conn = bind.contextual_connect() File "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect self._wrap_pool_connect(self.pool.connect, None), File "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect e, dialect, self) File "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1476, in _handle_dbapi_exception_noconnection exc_info File "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 265, in raise_from_cause reraise(type(exception), exception, tb=exc_tb, cause=cause) File "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect return fn() File "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect return _ConnectionFairy._checkout(self) File "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 791, in _checkout fairy = _ConnectionRecord.checkout(pool) File "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 532, in checkout rec = pool._do_get() File "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 1287, in _do_get return self._create_connection() File "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 350, in _create_connection return _ConnectionRecord(self) File "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 477, in __init__ self.__connect(first_connect_check=True) File "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 674, in __connect connection = pool._invoke_creator(self) File "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 106, in connect return dialect.connect(*cargs, **cparams) File "/usr/src/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 412, in connect return self.dbapi.connect(*cargs, **cparams) File "/usr/src/venv/local/lib/python2.7/site-packages/MySQLdb/__init__.py", line 86, in Connect return Connection(*args, **kwargs) File "/usr/src/venv/local/lib/python2.7/site-packages/MySQLdb/connections.py", line 204, in __init__ super(Connection, self).__init__(*args, **kwargs2) sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (2003, "Can't connect to MySQL server on 'orchmysql2ykgeqni.mysql.database.azure.com' (110)") (Background on this error at: [http://sqlalche.me/e/e3q8)]
>
> [2018-09-22 00:22:15,461] \{jobs.py:1425} ERROR - Executor reports task instance %s finished (%s) although the task says its %s. Was the task killed externally?
> [2018-09-22 00:22:15,475] \{models.py:1595} ERROR - Executor reports task instance %s finished (%s) although the task says its %s. Was the task killed externally?
> [2018-09-22 00:22:15,475] \{jobs.py:1435} ERROR - Cannot load the dag bag to handle failure for <TaskInstance: f75d1cca5b17165c0a495c1f_ff0adcc8_d8cd_4abe_a0a0_4992329d0443_1o8xqz.ACPComputeGateway_0_score 2018-09-22 00:00:00 [queued]>. Setting task to FAILED without callbacks or retries. Do you have enough resources?
>
> [2018-09-22 00:22:13,408] \{local_executor.py:50} ERROR - Failed to execute task Command 'exec bash -c 'airflow run f75d1cca5b17165c0a495c1f_ff0adcc8_d8cd_4abe_a0a0_4992329d0443_1o8xqz ACPComputeGateway_0_score 2018-09-22T00:00:00 --local -sd /usr/local/airflow/dags/F75D1CCA5B17165C0A495C1F/f75d1cca5b17165c0a495c1f_ff0adcc8_d8cd_4abe_a0a0_4992329d0443_1o8xqz.py'' returned non-zero exit status 1.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)