[GitHub] [airflow] eitanme commented on issue #23361: Scheduler crashes with psycopg2.errors.DeadlockDetected exception

[GitBox <gi...@apache.org>]

eitanme commented on issue #23361:
URL: https://github.com/apache/airflow/issues/23361#issuecomment-1166560652

   I'm also receiving this error with a single `LocalExecutor` scheduler running on Airflow 2.3.2.
   
   Has anyone found an effective workaround for this?
   
   ```
   2022-06-24 08:01:10,633] {scheduler_job.py:756} ERROR - Exception when executing SchedulerJob._run_scheduler_loop
   Traceback (most recent call last):
     File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1706, in _execute_context
       cursor, statement, parameters, context
     File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 716, in do_execute
       cursor.execute(statement, parameters)
   psycopg2.errors.DeadlockDetected: deadlock detected
   DETAIL:  Process 271343 waits for ShareLock on transaction 251928224; blocked by process 282010.
   Process 282010 waits for ShareLock on transaction 251928207; blocked by process 271343.
   HINT:  See server log for query details.
   CONTEXT:  while updating tuple (474,18) in relation "task_instance"
   
   
   The above exception was the direct cause of the following exception:
   
   Traceback (most recent call last):
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 739, in _execute
       self._run_scheduler_loop()
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 827, in _run_scheduler_loop
       num_queued_tis = self._do_scheduling(session)
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 909, in _do_scheduling
       callback_to_run = self._schedule_dag_run(dag_run, session)
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 1161, in _schedule_dag_run
       dag_run.schedule_tis(schedulable_tis, session)
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py", line 68, in wrapper
       return func(*args, **kwargs)
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/dagrun.py", line 1042, in schedule_tis
       .update({TI.state: State.SCHEDULED}, synchronize_session=False)
     File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3196, in update
       execution_options={"synchronize_session": synchronize_session},
     File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 1670, in execute
       result = conn._execute_20(statement, params or {}, execution_options)
     File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1520, in _execute_20
       return meth(self, args_10style, kwargs_10style, execution_options)
     File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 314, in _execute_on_connection
       self, multiparams, params, execution_options
     File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1399, in _execute_clauseelement
       cache_hit=cache_hit,
     File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1749, in _execute_context
       e, statement, parameters, cursor, context
     File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1930, in _handle_dbapi_exception
       sqlalchemy_exception, with_traceback=exc_info[2], from_=e
     File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
       raise exception
     File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1706, in _execute_context
       cursor, statement, parameters, context
     File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 716, in do_execute
       cursor.execute(statement, parameters)
   sqlalchemy.exc.OperationalError: (psycopg2.errors.DeadlockDetected) deadlock detected
   DETAIL:  Process 271343 waits for ShareLock on transaction 251928224; blocked by process 282010.
   Process 282010 waits for ShareLock on transaction 251928207; blocked by process 271343.
   HINT:  See server log for query details.
   CONTEXT:  while updating tuple (474,18) in relation "task_instance"
   
   [SQL: UPDATE task_instance SET state=%(state)s WHERE task_instance.dag_id = %(dag_id_1)s AND task_instance.run_id = %(run_id_1)s AND (task_instance.task_id, task_instance.map_index) IN ((%(param_1_1_1)s, %(param_1_1_2)s))]
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org