[GitHub] [airflow] argibbs commented on issue #23361: Scheduler crashes with psycopg2.errors.DeadlockDetected exception

[GitBox <gi...@apache.org>]

argibbs commented on issue #23361:
URL: https://github.com/apache/airflow/issues/23361#issuecomment-1167203464

   I'm also seeing this issue, albeit with a slightly different query:
   
   ```
   psycopg2.errors.DeadlockDetected: deadlock detected
   DETAIL:  Process 16440 waits for ShareLock on transaction 788648981; blocked by process 16481.
   Process 16481 waits for ShareLock on transaction 788648979; blocked by process 16440.
   HINT:  See server log for query details.
   CONTEXT:  while deleting tuple (0,25) in relation "dag"
   
   <snip stack same as posted by others>
   
   sqlalchemy.exc.OperationalError: (psycopg2.errors.DeadlockDetected) deadlock detected
   DETAIL:  Process 16440 waits for ShareLock on transaction 788648981; blocked by process 16481.
   Process 16481 waits for ShareLock on transaction 788648979; blocked by process 16440.
   HINT:  See server log for query details.
   CONTEXT:  while deleting tuple (0,25) in relation "dag"
   
   [SQL: DELETE FROM dag WHERE dag.dag_id IN (%(dag_id_1_1)s) RETURNING dag.dag_id]
   [parameters: {'dag_id_1_1': 'Pipeline.DAG_NAME_REDACTED'}]
   (Background on this error at: http://sqlalche.me/e/14/e3q8)
   ```
   
   It happens pretty much every time I delete a dag. Only way I've found around it is to browse the dag runs, delete all but one of them, then delete the dag.
   
   I'm running multiple schedulers, and `use_row_locking = True`. Postgres 13.1.
   
   Our version path was 2.2.4 -> 2.3.2 (we skipped 2.3.0 and 2.3.1 because of reasons). 2.2.4 is fine, 2.3.2 is not. Anecdotally, db load appears much higher in 2.3.2 - the box running postgres is pretty much flat out all the time, versus ~25% previously. I don't have hard numbers, because several things changed at once in our testing env (mea culpa), but I will at some point be upgrading another env, and I'll be watching closely.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org