You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/06/30 23:37:45 UTC
[GitHub] [airflow] jedcunningham opened a new pull request #16741: Fix race condition with dagrun callbacks
jedcunningham opened a new pull request #16741:
URL: https://github.com/apache/airflow/pull/16741
Instead of immediately sending callbacks to be processed, wait until
after we commit so the dagrun.end_date is guaranteed to be there when
the callback runs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham closed pull request #16741: Fix race condition with dagrun callbacks
Posted by GitBox <gi...@apache.org>.
jedcunningham closed pull request #16741:
URL: https://github.com/apache/airflow/pull/16741
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on pull request #16741: Fix race condition with dagrun callbacks
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on pull request #16741:
URL: https://github.com/apache/airflow/pull/16741#issuecomment-873104707
(still needs more test coverage I think, but would like to get a preliminary review before I spend more time on it)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #16741: Fix race condition with dagrun callbacks
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #16741:
URL: https://github.com/apache/airflow/pull/16741#discussion_r669241514
##########
File path: airflow/jobs/scheduler_job.py
##########
@@ -856,13 +857,18 @@ def _do_scheduling(self, session) -> int:
# But this would take care of the scenario when the Scheduler is restarted after DagRun is
# created and the DAG is deleted / renamed
try:
- self._schedule_dag_run(dag_run, session)
+ callback_to_run = self._schedule_dag_run(dag_run, session)
+ callback_tuples.append((dag_run, callback_to_run))
except SerializedDagNotFound:
self.log.exception("DAG '%s' not found in serialized_dag table", dag_run.dag_id)
continue
guard.commit()
+ # We will send the callbacks after we commit to ensure the context is up to date when it gets run
Review comment:
```suggestion
# Send the callbacks after we commit to ensure the context is up to date when it gets run
```
The “we will” part doesn’t sound right to me; the callbacks *are* being sent now, not some time in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham merged pull request #16741: Fix race condition with dagrun callbacks
Posted by GitBox <gi...@apache.org>.
jedcunningham merged pull request #16741:
URL: https://github.com/apache/airflow/pull/16741
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham closed pull request #16741: Fix race condition with dagrun callbacks
Posted by GitBox <gi...@apache.org>.
jedcunningham closed pull request #16741:
URL: https://github.com/apache/airflow/pull/16741
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on pull request #16741: Fix race condition with dagrun callbacks
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on pull request #16741:
URL: https://github.com/apache/airflow/pull/16741#issuecomment-889314249
Merged, failures are flacky tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #16741: Fix race condition with dagrun callbacks
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #16741:
URL: https://github.com/apache/airflow/pull/16741#issuecomment-877812018
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham closed pull request #16741: Fix race condition with dagrun callbacks
Posted by GitBox <gi...@apache.org>.
jedcunningham closed pull request #16741:
URL: https://github.com/apache/airflow/pull/16741
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org