You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/10/14 10:38:04 UTC
[GitHub] [airflow] ephraimbuddy opened a new pull request #18975: Row lock TI query in SchedulerJob._process_executor_events
ephraimbuddy opened a new pull request #18975:
URL: https://github.com/apache/airflow/pull/18975
Using multiple schedulers causes Deadlock in _process_executor_events.
This PR fixes it.
The error:
```
[2021-10-14 08:37:18,377: INFO/ForkPoolWorker-1] Celery task ID: 6e8e34d/persistence.py", line 995, in _emit_update_statements
statement, multiparams
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
return meth(self, multiparams, params)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1130, in _execute_clauseelement
distilled_params,
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1317, in _execute_context
e, statement, parameters, cursor, context
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1511, in _handle_dbapi_exception
sqlalchemy_exception, with_traceback=exc_info[2], from_=e
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1257, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 912, in do_executemany
cursor.executemany(statement, parameters)
sqlalchemy.exc.OperationalError: (psycopg2.errors.DeadlockDetected) deadlock detected
DETAIL: Process 35678 waits for ShareLock on transaction 150292; blocked by process 58800.
Process 58800 waits for ShareLock on transaction 150291; blocked by process 58799.
Process 58799 waits for ShareLock on transaction 150289; blocked by process 35678.
HINT: See server log for query details.
CONTEXT: while updating tuple (11,54) in relation "task_instance"
[SQL: UPDATE task_instance SET external_executor_id=%(external_executor_id)s WHERE task_instance.task_id = %(task_instance_task_id)s AND task_instance.dag_id = %(task_instance_dag_id)s AND task_instance.run_id = %(task_instance_run_id)s]
[parameters: ({'external_executor_id': 'fe1c200a-ca4a-49b2-b5f9-ea53d5ed0846', 'task_instance_task_id': 'test_sensor_15', 'task_instance_dag_id': 'airflow_bug', 'task_instance_run_id': 'manual__2021-10-14T08:17:18.387618+00:00'}, {'external_executor_id': '1ee3e79a-ceab-4215-94cb-100d8de73bad', 'task_instance_task_id': 'test_sensor_22', 'task_instance_dag_id': 'airflow_bug', 'task_instance_run_id': 'manual__2021-10-14T08:17:18.387618+00:00'}, {'external_executor_id': '8873162e-519f-4ef2-890c-d38754760c81', 'task_instance_task_id': 'test_sensor_24', 'task_instance_dag_id': 'airflow_bug', 'task_instance_run_id': 'manual__2021-10-14T08:17:18.387618+00:00'}, {'external_executor_id': '52d05186-5297-4079-b94a-132bef845673', 'task_instance_task_id': 'test_sensor_27', 'task_instance_dag_id': 'airflow_bug', 'task_instance_run_id': 'manual__2021-10-14T08:17:18.387618+00:00'}, {'external_executor_id': '1e47d3a0-4398-4f07-9ebd-73e16ffa02ec', 'task_instance_task_id': 'test_sensor_28', 'task_ins
tance_dag_id': 'airflow_bug', 'task_instance_run_id': 'manual__2021-10-14T08:17:18.387618+00:00'}, {'external_executor_id': '1c89c6e0-8cb6-437b-9ee5-56968bb993fc', 'task_instance_task_id': 'test_sensor_35', 'task_instance_dag_id': 'airflow_bug', 'task_instance_run_id': 'manual__2021-10-14T08:17:18.387618+00:00'})]
(Background on this error at: http://sqlalche.me/e/13/e3q8)
```
I tested the fix with 3 schedulers and it works.
How to reproduce:
Using 2 schedulers and celery executor, run this dag:
[dags.zip](https://github.com/apache/airflow/files/7345217/dags.zip)
One scheduler will stop with errors like the above after some time.
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #18975: Row lock TI query in SchedulerJob._process_executor_events
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #18975:
URL: https://github.com/apache/airflow/pull/18975#issuecomment-943415425
Looks green! BTW. We have LOT MORE GREEN nowadays after all those small flaky test fixes :D.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #18975: Row lock TI query in SchedulerJob._process_executor_events
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #18975:
URL: https://github.com/apache/airflow/pull/18975#discussion_r728906466
##########
File path: airflow/jobs/scheduler_job.py
##########
@@ -510,7 +510,15 @@ def _process_executor_events(self, session: Session = None) -> int:
# Check state of finished tasks
filter_for_tis = TI.filter_for_tis(tis_with_right_state)
- tis: List[TI] = session.query(TI).filter(filter_for_tis).options(selectinload('dag_model')).all()
+ query = session.query(TI).filter(filter_for_tis).options(selectinload('dag_model'))
+ # row lock this entire set of tasks to make sure the scheduler doesn't fail when we have
+ # multi-schedulers
+ tis: List[TI] = with_row_locks(
+ query,
+ of=TI,
+ session=session,
+ **skip_locked(session=session),
+ ).all()
Review comment:
```suggestion
tis: Iterator[TI] = with_row_locks(
query,
of=TI,
session=session,
**skip_locked(session=session),
)
```
Probably a good time to change this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a change in pull request #18975: Row lock TI query in SchedulerJob._process_executor_events
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #18975:
URL: https://github.com/apache/airflow/pull/18975#discussion_r728910465
##########
File path: airflow/jobs/scheduler_job.py
##########
@@ -510,7 +510,15 @@ def _process_executor_events(self, session: Session = None) -> int:
# Check state of finished tasks
filter_for_tis = TI.filter_for_tis(tis_with_right_state)
- tis: List[TI] = session.query(TI).filter(filter_for_tis).options(selectinload('dag_model')).all()
+ query = session.query(TI).filter(filter_for_tis).options(selectinload('dag_model'))
+ # row lock this entire set of tasks to make sure the scheduler doesn't fail when we have
+ # multi-schedulers
+ tis: List[TI] = with_row_locks(
+ query,
+ of=TI,
+ session=session,
+ **skip_locked(session=session),
+ ).all()
Review comment:
Actually yeah - now when we do "for update" for those rows, changing to iterator makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #18975: Row lock TI query in SchedulerJob._process_executor_events
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #18975:
URL: https://github.com/apache/airflow/pull/18975#issuecomment-943243841
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #18975: Row lock TI query in SchedulerJob._process_executor_events
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #18975:
URL: https://github.com/apache/airflow/pull/18975#issuecomment-943449013
> Looks green! BTW. We have LOT MORE GREEN nowadays after all those small flaky test fixes :D.
Yay!! ππ Thanks for the good worksππ
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy merged pull request #18975: Row lock TI query in SchedulerJob._process_executor_events
Posted by GitBox <gi...@apache.org>.
ephraimbuddy merged pull request #18975:
URL: https://github.com/apache/airflow/pull/18975
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org