You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "dstandish (via GitHub)" <gi...@apache.org> on 2024/04/10 21:18:44 UTC
[PR] Don't wait for lock on DagRun in mini scheduler [airflow]
dstandish opened a new pull request, #38914:
URL: https://github.com/apache/airflow/pull/38914
We should just bail if something else is already "minischeduling". We already catch OperationalError so there's nothing else we need to change for this.
@collinmcnulty and @jedcunningham you might be interested in this. Discovered in collaboration with @seanmuth that this can cause an issue if many tasks from same dag run are trying to "minischedule" at the same time. E.g. task mapping. If something else is already locking, we should just move on and not wait.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Don't wait for lock on DagRun in mini scheduler [airflow]
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish merged PR #38914:
URL: https://github.com/apache/airflow/pull/38914
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Don't wait for lock on DagRun in mini scheduler [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on PR #38914:
URL: https://github.com/apache/airflow/pull/38914#issuecomment-2048845639
This might have the inverse effect of putting more load on the scheduler as it might end up doing more task expansion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org