You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "wolfier (via GitHub)" <gi...@apache.org> on 2025/01/03 23:44:13 UTC
[I] HA Scheduler does not respect max_active_runs in edge case [airflow]
wolfier opened a new issue, #45388:
URL: https://github.com/apache/airflow/issues/45388
### Apache Airflow version
2.10.4
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
Two queued dagruns of a DAG with max_active_runs of 1 started within 0.2 seconds of each other.
The deployment has two schedulers, A and B. I suspect scheduler A started one dagrun and scheduler B started the other dagrun. Because a scheduler queries the active dagrun information every scheduling loop via [_start_queued_dagruns]( https://github.com/apache/airflow/blob/2.10.4/airflow/jobs/scheduler_job_runner.py#L1537), it is possible for the limit to be exceeded as the information is not shared between schedulers. Both schedulers thought there were no active dagruns and started their respective queued dagrun.
The question is how does one dagrun end up in one query and not the other. One explanation could be scheduler A goes out and locks only 1 dagrun row because the other dagrun row is out of the [max_dagruns_per_loop_to_schedule](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#max-dagruns-per-loop-to-schedule) range. The other scheduler then picked up the dagrun that did not get queried. Even though it is very unlikely, I suspect both scheduling loops ran the query very closely in time.
### What you think should happen instead?
_No response_
### How to reproduce
This scenario requires extreme luck (or lack thereof) so I have not been able to reproduce this behaviour.
Perhaps the key to reproduce this is with max_dagruns_per_loop_to_schedule set to 1.
### Operating System
n/a
### Versions of Apache Airflow Providers
_No response_
### Deployment
Astronomer
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] HA Scheduler does not respect max_active_runs in edge case [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #45388:
URL: https://github.com/apache/airflow/issues/45388#issuecomment-2571281856
Yes. That is quite possible edge case though rather infrequent. I am not sure if there is a way we could protect it - we would likely have to add lock on dag not only on DagRun, but it would likely heavily decrease some parallelism scenarios for scheduling. @ashb wdyt ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] HA Scheduler does not respect max_active_runs in edge case [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #45388:
URL: https://github.com/apache/airflow/issues/45388#issuecomment-3713417732
This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activity on this issue. Kindly asking to recheck the report against latest Airflow version and let us know if the issue is reproducible. The issue will be closed in next 30 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] HA Scheduler does not respect max_active_runs in edge case [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #45388: HA Scheduler does not respect max_active_runs in edge case
URL: https://github.com/apache/airflow/issues/45388
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] HA Scheduler does not respect max_active_runs in edge case [airflow]
Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on issue #45388:
URL: https://github.com/apache/airflow/issues/45388#issuecomment-2571407550
As I was chatting with Alan about this yesterday, another thought I had was to do optimistic locking. e.g. set the state to running only if the count of running for that dag matches what we expect (and used to determine it's okay to start another!).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] HA Scheduler does not respect max_active_runs in edge case [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #45388:
URL: https://github.com/apache/airflow/issues/45388#issuecomment-2571411707
Generally:
a) pessimistic locking
* lock update
*run query and update state
* save the state change and unlocki
b) optimistic locking
* read a state
* run query and only update if the state has not changed in a single transaction and commit in the same operation
I am not sure what "state" we can read here and how we can verify the state has not changed sinve we read it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] HA Scheduler does not respect max_active_runs in edge case [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #45388:
URL: https://github.com/apache/airflow/issues/45388#issuecomment-2571410913
Can we do it ? How?
I think this is the "logical" problem to solve.
Whatever optimisitc query you can run is "read" and in order to make it really protected, you need to make it part of a transaction - you need to query and update the state in the same transaction essentially. Or you have to read a "state" of the system and use it in the "write" statement to detect that the state has changed. You cannot really separate "read" and "update" action - they should be connected by either transaction or state retrieved from the "read" action has to be used as a "potentially failing" check in the update state.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] HA Scheduler does not respect max_active_runs in edge case [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #45388:
URL: https://github.com/apache/airflow/issues/45388#issuecomment-3851490162
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org