You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/11/10 08:47:54 UTC

[GitHub] [airflow] tlw-ray opened a new issue #12234: DAG success bug Tasks in it no_status

tlw-ray opened a new issue #12234:
URL: https://github.com/apache/airflow/issues/12234


   **Apache Airflow version**: 1.10.12
   
   
   **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): none
   
   **Environment**:
   
   - **Cloud provider or hardware configuration**: 
   - **OS** (e.g. from /etc/os-release): 
       - NAME="CentOS Linux"
       - VERSION="7 (Core)"
       - ID="centos"
       - ID_LIKE="rhel fedora"
       - VERSION_ID="7"
       - PRETTY_NAME="CentOS Linux 7 (Core)"
   - **Kernel** (e.g. `uname -a`):Linux dev-dbmid-161 3.10.0-957.10.1.el7.x86_64 #1 SMP Mon Mar 18 15:06:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
   - **Install tools**: https://github.com/puckel/docker-airflow 
   - **Others**: Run by docker-compose, use CeleryExecutor, redis, postgres.
   
   **What happened**:
   
   Airflow runs for about a week and is scheduled once a day (DAG status is success, Task status is success). However, occasionally the DAG status is success, but the Tasks are no_status. That is to say, the DAG returns a successful status as soon as it is executed, but none of the tasks in it are running.
   
   <img width="755" alt="Airflow_20201110160955" src="https://raw.githubusercontent.com/tlw-ray/docker-airflow-mssql/master/image/Airflow_20201110160955.png">
   
   <!-- (please include exact error messages if you can) -->
   
   **What you expected to happen**:
   
   Tasks in the two `DAGs` are in `no_status` and have not been executed. It is expected that these tasks should be executed and be in the state of `success` like other `DAG` execution results.
   
   <!-- What do you think went wrong? -->
   
   **How to reproduce it**:
   
   If the DAG is triggered, it can always be executed correctly, but occasionally this happens unexpectedly during scheduling.
   
   
   **Anything else we need to know**:
   
   my dag: 
   
   ~~~python
   dt = datetime.now() - timedelta(days=1)
   
   default_args = {
     'start_date': datetime(dt.year, dt.month, dt.day, 19),
   }
   
   dag = DAG(
       dag_id='DATA_FLOW',
       default_args=default_args,
       schedule_interval=timedelta(days=1),
   )
   ~~~


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #12234: DAG success bug Tasks in it no_status

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #12234:
URL: https://github.com/apache/airflow/issues/12234#issuecomment-724556136


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ashb closed issue #12234: DAG success bug Tasks in it no_status

Posted by GitBox <gi...@apache.org>.
ashb closed issue #12234:
URL: https://github.com/apache/airflow/issues/12234


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] tlw-ray commented on issue #12234: DAG success bug Tasks in it no_status

Posted by GitBox <gi...@apache.org>.
tlw-ray commented on issue #12234:
URL: https://github.com/apache/airflow/issues/12234#issuecomment-731947810


   It can be seen that once every five minutes, every twelve runs, there will be a problem exactly one hour. If the interval time is changed to three minutes, such a problem will not occur.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] tlw-ray commented on issue #12234: DAG success bug Tasks in it no_status

Posted by GitBox <gi...@apache.org>.
tlw-ray commented on issue #12234:
URL: https://github.com/apache/airflow/issues/12234#issuecomment-725230565


   The problem is very easy to reproduce:
   ~~~python
   from builtins import range
   from datetime import datetime, timedelta
   
   import airflow
   from airflow.models import DAG
   from airflow.operators.bash_operator import BashOperator
   from airflow.operators.dummy_operator import DummyOperator
   
   dt = datetime.now()
   
   args = {
       'start_date': datetime(dt.year, dt.month, dt.day, dt.hour),
   }
   
   dag = DAG(
       dag_id='5分钟一跑测试稳定性',
       default_args=args,
       schedule_interval=timedelta(minutes=5),
   )
   
   
   run_this = BashOperator(
       task_id='pwdTask',
       bash_command='pwd ',
       dag=dag,
   )
   ~~~
   
   ![](https://raw.githubusercontent.com/tlw-ray/docker-airflow-mssql/master/image/Airflow_1_10_12.png)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ashb commented on issue #12234: DAG success bug Tasks in it no_status

Posted by GitBox <gi...@apache.org>.
ashb commented on issue #12234:
URL: https://github.com/apache/airflow/issues/12234#issuecomment-733913202


   ```
   dt = datetime.now()
   
   args = {
       'start_date': datetime(dt.year, dt.month, dt.day, dt.hour),
   }
   ```
   
   This is the source of your problem. When it comes to evaluate the task, the start date is in the future, so the scheduler skips it.
   
   Use a fixed start date and set catchup=False


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org