You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/06/04 01:29:17 UTC

[GitHub] [airflow] ChristianYeah opened a new issue #9131: max_active_runs failed in production env with multiple workers

ChristianYeah opened a new issue #9131:
URL: https://github.com/apache/airflow/issues/9131

**Apache Airflow version**: 1.10.10

**Environment**:

- **Cloud provider or hardware configuration**:
- **OS** (e.g. from /etc/os-release): centos 7.5
- **Kernel** (e.g. `uname -a`):Linux prod-auto-ops001 3.10.0-1062.4.1.el7.x86_64 #1 SMP Fri Oct 18 17:15:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
- **Install tools**: pip
- **Others**:

**What happened**:

I have two environments, one is development, another is production. They uses CeleryExecutor and redis for broker, mysql for result backend and the airflow db. The only difference is production env got two worker.

I passed the params, max_active_runs in my dag defination , eg:
```
dag = DAG(
dag_id='test_max_active_runs',
default_args=default_args,
schedule_interval=None,
start_date=airflow.utils.dates.days_ago(1),
max_active_runs=1
)
```
In the dev env, when the task is scheduled, the dashboard will show 1/1 active dag runs in red color, while nothing happens in the production env.

May I suppose that the setting max_active_runs is not working in my production env?

**What you expected to happen**:

It should be exactly the same in the production env comparison to the dev env.

**How to reproduce it**:
Let me know if you need further information

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ChristianYeah commented on issue #9131: max_active_runs failed in production env with multiple workers

Posted by GitBox <gi...@apache.org>.

ChristianYeah commented on issue #9131:
URL: https://github.com/apache/airflow/issues/9131#issuecomment-641222683


   I’ll duplicate the production env to the test later later , see if it will be reproduced.&nbsp;
   
   
   
   发自我的iPhone
   
   
   ------------------ Original ------------------
   From: svetli-n <notifications@github.com&gt;
   Date: Tue,Jun 9,2020 7:12 PM
   To: apache/airflow <airflow@noreply.github.com&gt;
   Cc: ChristianYe <ye56119537@vip.qq.com&gt;, Mention <mention@noreply.github.com&gt;
   Subject: Re: [apache/airflow] max_active_runs failed in production env with multiple workers (#9131)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] svetli-n commented on issue #9131: max_active_runs failed in production env with multiple workers

Posted by GitBox <gi...@apache.org>.

svetli-n commented on issue #9131:
URL: https://github.com/apache/airflow/issues/9131#issuecomment-639437006


   Have you scripted building your envs, eg can you provide docker-compose files?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] svetli-n commented on issue #9131: max_active_runs failed in production env with multiple workers

Posted by GitBox <gi...@apache.org>.

svetli-n commented on issue #9131:
URL: https://github.com/apache/airflow/issues/9131#issuecomment-641221074


   @ChristianYeah Thanks. I can't reproduce it though. Feel free to share full dag run file including deafult settings and tasks, as well as how you trigger the dag run.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #9131: max_active_runs failed in production env with multiple workers

Posted by GitBox <gi...@apache.org>.

eladkal commented on issue #9131:
URL: https://github.com/apache/airflow/issues/9131#issuecomment-921708891


   This issue is reported against older version of Airflow.
   Please check against latest version. If needed we will reopen this issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #9131: max_active_runs failed in production env with multiple workers

Posted by GitBox <gi...@apache.org>.

boring-cyborg[bot] commented on issue #9131:
URL: https://github.com/apache/airflow/issues/9131#issuecomment-638547111


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ChristianYeah commented on issue #9131: max_active_runs failed in production env with multiple workers

Posted by GitBox <gi...@apache.org>.

ChristianYeah commented on issue #9131:
URL: https://github.com/apache/airflow/issues/9131#issuecomment-655984064


   > @ChristianYeah Thanks. I can't reproduce it though. Feel free to share full dag run file including deafult settings and tasks, as well as how you trigger the dag run.
   
   Hi, @svetli-n , sorry for the late reponse, here is the code
   
   ```
   # coding:utf-8
   import airflow
   from airflow import DAG
   from airflow.operators.dummy_operator import DummyOperator
   from airflow.operators.python_operator import PythonOperator
   from fabric import Connection
   
   default_args = {
       'owner': 'yexiaodong',
   }
   
   dag = DAG(
       dag_id='sync_dags',
       default_args=default_args,
       schedule_interval=None,
       start_date=airflow.utils.dates.days_ago(1),
       max_active_runs=1,
   )
   
   
   def git_pull_function():
       for host in ["172.16.16.178", "172.16.16.177"]:
           with Connection(host=host, user='ops', port=22) as c:
               result = c.run('cd /home/ops/code/master_dag_repo && git pull')
               print(host, result)
   
   
   git_pull = PythonOperator(
       task_id='git-pull',
       python_callable=git_pull_function,
       dag=dag,
   )
   
   ```
   
   I would like to limit the max active dag run to 1 for ensuring the sync progress won't go wrong. It works on the local env, but the production env seems like not working as expected
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal closed issue #9131: max_active_runs failed in production env with multiple workers

Posted by GitBox <gi...@apache.org>.

eladkal closed issue #9131:
URL: https://github.com/apache/airflow/issues/9131


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ChristianYeah commented on issue #9131: max_active_runs failed in production env with multiple workers

Posted by GitBox <gi...@apache.org>.

ChristianYeah commented on issue #9131:
URL: https://github.com/apache/airflow/issues/9131#issuecomment-640383392


   > Have you scripted building your envs, eg can you provide docker-compose files?
   
   @svetli-n , I installed everything via pip (pip install apache-airflow[all]), except the local virtual env was installed by anaconda and the production was installed from source code. additionally, except the db or broker config, everything is the same.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org