You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/03/11 21:11:40 UTC
[GitHub] [airflow] stas-snow opened a new issue #22193: max_active_runs = 1 can still create multiple active execution runs (again)
stas-snow opened a new issue #22193:
URL: https://github.com/apache/airflow/issues/22193
### Apache Airflow version
2.2.3
### What happened
Same issue as reported in: #9975 - max_active_runs** = 1 can still create multiple active execution runs.
I'm seeing this issue in airflow 2.2.3.
catchup=True and max_active_runs=1.
DAG is triggered multiple times and multiple instances are running in parallel.
### What you expected to happen
Expect that only one instance of the DAG is running at any given time if max_active_runs=1.
### How to reproduce
_No response_
### Operating System
GCP: composer-2.0.4-airflow-2.2.3
### Versions of Apache Airflow Providers
https://cloud.google.com/composer/docs/concepts/versioning/composer-versions
composer-2.0.4-airflow-2.2.3
### Deployment
Composer
### Deployment details
composer-2.0.4-airflow-2.2.3
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on issue #22193: max_active_runs = 1 can still create multiple active execution runs (again)
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on issue #22193:
URL: https://github.com/apache/airflow/issues/22193#issuecomment-1065542561
Describe how to reproduce it including a sample dag to reproduce it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on issue #22193: max_active_runs = 1 can still create multiple active execution runs (again)
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on issue #22193:
URL: https://github.com/apache/airflow/issues/22193#issuecomment-1067883017
Your DAG has `max_active_runs` in the `default_args` instead of being on the dag itself. `default_args` is for tasks.
Set the max_active_runs like this:
```
@dag(default_args=default_dag_args, max_active_runs=1)
def test2_dag():
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy closed issue #22193: max_active_runs = 1 can still create multiple active execution runs (again)
Posted by GitBox <gi...@apache.org>.
ephraimbuddy closed issue #22193:
URL: https://github.com/apache/airflow/issues/22193
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #22193: max_active_runs = 1 can still create multiple active execution runs (again)
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #22193:
URL: https://github.com/apache/airflow/issues/22193#issuecomment-1065527108
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] stas-snow commented on issue #22193: max_active_runs = 1 can still create multiple active execution runs (again)
Posted by GitBox <gi...@apache.org>.
stas-snow commented on issue #22193:
URL: https://github.com/apache/airflow/issues/22193#issuecomment-1067079346
Code:
```
import logging
import time
# Airflow packages
from random import uniform
from airflow.decorators import dag, task
logger = logging.getLogger(__name__)
default_dag_args = {
'owner': 'airflow',
'start_date': '2022-03-01',
'schedule_interval': '@daily',
'retries': 0,
'catchup': True,
'max_active_runs': 1,
'depends_on_past': False,
}
@dag(default_args=default_dag_args)
def test2_dag():
@task()
def start():
logging.info("Starting DAG. Going to sleep")
time.sleep(uniform(1, 5) * 60)
logging.info("First task, exiting")
return None
@task()
def do():
time.sleep(uniform(1, 5) * 60)
return None
@task()
def end():
logging.info(f"Finishing...")
time.sleep(3)
return None
# Define dependency between tasks
start() >> do() >> end()
# start the DAG
test_pipeline = test2_dag()
```
Screenshot of multiple DAG instances all in "Running" state.
<img width="546" alt="image" src="https://user-images.githubusercontent.com/93541868/158224423-fae9b635-abd2-4be5-a292-fa2c6d3c5398.png">
Documentation says that there should not be more than max_active_runs in Running state:
```
max_active_runs – maximum number of active DAG runs, beyond this number of DAG runs in a running state, the scheduler won’t create new active DAG runs
```
I would expect only 1 DAG in Running state if I have "max_active_runs = 1".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org