You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jack (JIRA)" <ji...@apache.org> on 2018/12/17 08:23:00 UTC
[jira] [Commented] (AIRFLOW-2660) Deadlock error from SQLAlchemy
when the parallelism is increased.
[ https://issues.apache.org/jira/browse/AIRFLOW-2660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16722777#comment-16722777 ]
jack commented on AIRFLOW-2660:
-------------------------------
I suspect that this is a known MySQL issue:
[https://stackoverflow.com/questions/41015813/avoiding-mysql-deadlock-when-upgrading-shared-to-exclusive-lock]
> Deadlock error from SQLAlchemy when the parallelism is increased.
> -----------------------------------------------------------------
>
> Key: AIRFLOW-2660
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2660
> Project: Apache Airflow
> Issue Type: Bug
> Components: db
> Affects Versions: 1.9.0
> Environment: RHEL 7
> Reporter: Vignesh
> Priority: Minor
> Labels: patch
> Fix For: 1.9.0
>
>
> Faced the following errors when running a multi worker distributed setup of airflow 1.9. These error occurred when the number of scheduler threads, parallelism and dag_concurrency were increased in the configuration.
>
> Original exception was: (_mysql_exceptions.OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') [SQL: u'UPDATE task_instance SET state=%s WHERE task_instance.task_id = %s AND task_instance.dag_id = %s AND task_instance.execution_date = %s'] [parameters: (u'queued', 'join', 'parent_dag.sub_dag', datetime.datetime(2018, 6, 18, 7, 1, 44))] (Background on this error at: [http://sqlalche.me/e/e3q8])
>
> sqlalchemy.exc.InvalidRequestError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (_mysql_exceptions.OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') [SQL: u'UPDATE task_instance SET state=%s WHERE task_instance.task_id = %s AND task_instance.dag_id = %s AND task_instance.execution_date = %s'] [parameters: (u'queued', 'join_0', 'parent_dag.sub_dag', datetime.datetime(2018, 6, 18, 7, 1, 44))] (Background on this error at: [http://sqlalche.me/e/e3q8])
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)