[GitHub] [airflow] Airflowcode commented on pull request #14031: Disable row level locking for Mariadb and MySQL <8

[GitBox <gi...@apache.org>]

Airflowcode commented on PR #14031:
URL: https://github.com/apache/airflow/pull/14031#issuecomment-1179193809

   We have a similar issue with Airflow 2.3.1 and MySQL 'Ver 14.14 Distrib 5.7.37'. Intermittently we get Deadlock errors as shown below, immediately after these Deadlocks Schulder is shutting down or waiting for the running job to complete. Is there any temporary solution or workaround for this issue? Any suggestion for changing the Airflow version to overcome this Deadlock issue? 
   
   
   
    File "/efs/airflow-venv/cdw-prod/airflow/lib/python3.9/site-packages/MySQLdb/cursors.py", line 239, in executemany
       self.rowcount = sum(self.execute(query, arg) for arg in args)
     File "/efs/airflow-venv/cdw-prod/airflow/lib/python3.9/site-packages/MySQLdb/cursors.py", line 239, in <genexpr>
       self.rowcount = sum(self.execute(query, arg) for arg in args)
     File "/efs/airflow-venv/cdw-prod/airflow/lib/python3.9/site-packages/MySQLdb/cursors.py", line 206, in execute
       res = self._query(query)
     File "/efs/airflow-venv/cdw-prod/airflow/lib/python3.9/site-packages/MySQLdb/cursors.py", line 319, in _query
       db.query(q)
     File "/efs/airflow-venv/cdw-prod/airflow/lib/python3.9/site-packages/MySQLdb/connections.py", line 254, in query
       _mysql.connection.query(self, query)
   sqlalchemy.exc.OperationalError: (MySQLdb._exceptions.OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction')
   [SQL: UPDATE dag_run SET last_scheduling_decision=%s WHERE dag_run.id = %s]
   [parameters: ((datetime.datetime(2022, 7, 8, 11, 0, 23, 20492), 4611), (datetime.datetime(2022, 7, 8, 11, 0, 23, 92501), 4612), (datetime.datetime(2022, 7, 8, 11, 0, 23, 180288), 4613), (datetime.datetime(2022, 7, 8, 11, 0, 23, 261947), 4616), (datetime.datetime(2022, 7, 8, 11, 0, 23, 360666), 4619), (datetime.datetime(2022, 7, 8, 11, 0, 23, 484052), 4620), (datetime.datetime(2022, 7, 8, 11, 0, 23, 605315), 4621), (datetime.datetime(2022, 7, 8, 11, 0, 23, 696649), 4622)  ... displaying 10 of 17 total bound parameter sets ...  (datetime.datetime(2022, 7, 8, 11, 0, 22, 901407), 4632), (datetime.datetime(2022, 7, 8, 11, 0, 22, 964056), 4633))]
   (Background on this error at: http://sqlalche.me/e/14/e3q8)
   [^[[^[[0m] {^[[34mlocal_executor.py:^[[0m390} INFO^[[0m - Shutting down LocalExecutor; waiting for running tasks to finish.  Signal again if you don't want to wait.^[[0m
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org