You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/09/21 16:45:44 UTC
[GitHub] [airflow] denysivanov opened a new issue #18409: Setting max_active_runs = 1 - still have multiple instances of the same dag running
denysivanov opened a new issue #18409:
URL: https://github.com/apache/airflow/issues/18409
### Apache Airflow version
2.1.3
### Operating System
Linux
### Versions of Apache Airflow Providers
apache-airflow-providers-microsoft-azure==3.1.1
apache-airflow-providers-microsoft-mssql==2.0.1
apache-airflow-providers-odbc==2.0.1
apache-airflow-providers-slack==4.0.1
### Deployment
Other 3rd-party Helm chart
### Deployment details
We are using this helm chart
https://github.com/airflow-helm/charts/tree/main/charts/airflow
### What happened
I set max_active_runs=1 and still see multiple instances of the same dag running
with DAG(
dag_id=dag_helper.get_dag_id_from_path(__file__),
default_args=dag_helper.get_dag_default_args(),
schedule_interval=dag_helper.set_schedule_interval(prod_interval="30 13,16,19 * * *", qa_interval="*/5 * * * *"),
max_active_runs=1,
start_date=days_ago(2),
tags=dag_helper.get_tags(__file__)) as dag:
inbound_dag_flat_file_processor(inbound_dag_config
)
### What you expected to happen
Should have only one active running instance.
New instance of dag should start in the queued state.
According this this one https://github.com/apache/airflow/pull/16401 this problem should be fixed in v2.1.3
### How to reproduce
We start instance of dag inside of the current dag
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Jorricks commented on issue #18409: Setting max_active_runs = 1 - still have multiple instances of the same dag running
Posted by GitBox <gi...@apache.org>.
Jorricks commented on issue #18409:
URL: https://github.com/apache/airflow/issues/18409#issuecomment-924615761
Hello @denysivanov,
In the example code you submitted, you create a DagRun with status Running. However, this is done directly on the database and thereby you could consider this to be a manual overwrite.
The limits, max_active_runs, are enforced by the scheduler component of Airflow. This component will create DagRuns based on the schedule and put them in the queued state first. It will only mark them as running if that will not surpass any thresholds like the max_active_runs.
By completely skipping those scheduler steps, there is no way for Airflow to enforce the limit on max_active_runs. Hence, I propose you change the code you showed to set the DagRun to Queued state instead of Running. That should fix your issue :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] denysivanov commented on issue #18409: Setting max_active_runs = 1 - still have multiple instances of the same dag running
Posted by GitBox <gi...@apache.org>.
denysivanov commented on issue #18409:
URL: https://github.com/apache/airflow/issues/18409#issuecomment-926061357
@Jorricks it is working in QA will try in PROD now
Appreciate your help here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] denysivanov commented on issue #18409: Setting max_active_runs = 1 - still have multiple instances of the same dag running
Posted by GitBox <gi...@apache.org>.
denysivanov commented on issue #18409:
URL: https://github.com/apache/airflow/issues/18409#issuecomment-925332596
@Jorricks doing it now!
Will update you on status tomorrow September 23, 2021.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy closed issue #18409: Setting max_active_runs = 1 - still have multiple instances of the same dag running
Posted by GitBox <gi...@apache.org>.
ephraimbuddy closed issue #18409:
URL: https://github.com/apache/airflow/issues/18409
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #18409: Setting max_active_runs = 1 - still have multiple instances of the same dag running
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #18409:
URL: https://github.com/apache/airflow/issues/18409#issuecomment-924166462
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on issue #18409: Setting max_active_runs = 1 - still have multiple instances of the same dag running
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on issue #18409:
URL: https://github.com/apache/airflow/issues/18409#issuecomment-925338165
Also @denysivanov , upgrade to 2.1.4, there’s a bug on max_active_runs that have been fixed. Not related to your issue though. @Jorricks answered you correctly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org