You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/03/02 18:53:25 UTC
[GitHub] [airflow] imbaczek opened a new issue #21947: "deadlock detected" error
imbaczek opened a new issue #21947:
URL: https://github.com/apache/airflow/issues/21947
### Apache Airflow version
2.0.2
### What happened
```
2022-02-26 15:42:51 UTC [104192]: user=prod_airflow,db=prod_airflow,app=[unknown],client=10.80.166.208 STATEMENT: UPDATE dag SET next_dagrun='2022-02-26T15:32:50.428672+00:00'::timestamptz, next_dagrun_create_after='2022-02-26T15:42:50.428672+00:00'::timestamptz WHERE dag.dag_id = 'redacted_1'
2022-02-26 15:43:07 UTC [130186]: user=prod_airflow,db=prod_airflow,app=[unknown],client=10.80.166.208 ERROR: deadlock detected
2022-02-26 15:43:07 UTC [130186]: user=prod_airflow,db=prod_airflow,app=[unknown],client=10.80.166.208 DETAIL: Process 130186 waits for ShareLock on transaction 1728784371; blocked by process 130573.
Process 130573 waits for ShareLock on transaction 1728784356; blocked by process 130186.
Process 130186: UPDATE dag SET next_dagrun='2022-02-26T15:42:47.604016+00:00'::timestamptz, next_dagrun_create_after='2022-02-26T15:52:47.604016+00:00'::timestamptz WHERE dag.dag_id = 'redacted_1'
Process 130573: UPDATE dag SET last_parsed_time='2022-02-26T15:43:03.633626+00:00'::timestamptz, next_dagrun='2022-02-26T15:40:03.633649+00:00'::timestamptz WHERE dag.dag_id = 'redacted_2'
2022-02-26 15:43:07 UTC [130186]: user=prod_airflow,db=prod_airflow,app=[unknown],client=10.80.166.208 HINT: See server log for query details.
2022-02-26 15:43:07 UTC [130186]: user=prod_airflow,db=prod_airflow,app=[unknown],client=10.80.166.208 CONTEXT: while updating tuple (4,2) in relation "dag"
2022-02-26 15:43:07 UTC [130186]: user=prod_airflow,db=prod_airflow,app=[unknown],client=10.80.166.208 STATEMENT: UPDATE dag SET next_dagrun='2022-02-26T15:42:47.604016+00:00'::timestamptz, next_dagrun_create_after='2022-02-26T15:52:47.604016+00:00'::timestamptz WHERE dag.dag_id = 'redacted_1'
```
Likely places which conflict in the code:
https://github.com/apache/airflow/blob/d7265791187fb2117dfd090cdb7cce3f8c20866c/airflow/models/dag.py#L2436
https://github.com/apache/airflow/blob/d7265791187fb2117dfd090cdb7cce3f8c20866c/airflow/models/dag.py#L2919
### What you expected to happen
_No response_
### How to reproduce
We have 830 dags and this number keeps growing due to business requirements. No special steps to reproduce are required, it just happens every now and then, as would be expected. Increase fillfactor, perhaps?
### Operating System
VERSION="20.04.3 LTS (Focal Fossa)"
### Versions of Apache Airflow Providers
apache-airflow-providers-celery==1.0.1
apache-airflow-providers-ftp==1.0.1
apache-airflow-providers-http==1.1.0
apache-airflow-providers-imap==1.0.1
apache-airflow-providers-sqlite==1.0.1
### Deployment
Other Docker-based deployment
### Deployment details
```
$ docker --version
Docker version 19.03.5, build 633a0ea838
```
Postgres: psql (11.5 (Ubuntu 11.5-1.pgdg16.04+1))
### Anything else
Multiple times a day.
Took remediation steps to reduce amount of rows on page:
```
prod_airflow=# alter table dag set (fillfactor = 10);
ALTER TABLE
prod_airflow=# vacuum full dag;
VACUUM
```
which reduced frequency to ~1/day.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on issue #21947: "deadlock detected" error in dag table updates
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on issue #21947:
URL: https://github.com/apache/airflow/issues/21947#issuecomment-1059078557
I have seen this error with multi-scheduler setup. How many schedulers are you using?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on issue #21947: "deadlock detected" error in dag table updates
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on issue #21947:
URL: https://github.com/apache/airflow/issues/21947#issuecomment-1059294902
Deadlocks are fine except if they caused the scheduler to stop. From the log above, this happens in your database and does not stop your scheduler. Looks similar to this fix https://github.com/apache/airflow/pull/19842
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] imbaczek commented on issue #21947: "deadlock detected" error in dag table updates
Posted by GitBox <gi...@apache.org>.
imbaczek commented on issue #21947:
URL: https://github.com/apache/airflow/issues/21947#issuecomment-1059285594
Just one active scheduler at a time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #21947: "deadlock detected" error in dag table updates
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #21947:
URL: https://github.com/apache/airflow/issues/21947#issuecomment-1060064887
Yep. duplicate of #19842
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21947: "deadlock detected" error in dag table updates
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #21947:
URL: https://github.com/apache/airflow/issues/21947#issuecomment-1060064887
Yep. duplicagte of #19842
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #21947: "deadlock detected" error
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #21947:
URL: https://github.com/apache/airflow/issues/21947#issuecomment-1057269511
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #21947: "deadlock detected" error in dag table updates
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #21947:
URL: https://github.com/apache/airflow/issues/21947
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org