You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Bolke de Bruin (JIRA)" <ji...@apache.org> on 2017/03/14 19:27:41 UTC
[jira] [Resolved] (AIRFLOW-934) airflow delayed the task to start
[ https://issues.apache.org/jira/browse/AIRFLOW-934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bolke de Bruin resolved AIRFLOW-934.
------------------------------------
Resolution: Won't Fix
1.6.1. is an unsupported version. Report back if same issue exists on 1.8.0 (rc5)
> airflow delayed the task to start
> ----------------------------------
>
> Key: AIRFLOW-934
> URL: https://issues.apache.org/jira/browse/AIRFLOW-934
> Project: Apache Airflow
> Issue Type: Bug
> Components: DagRun
> Affects Versions: Airflow 1.6.2
> Reporter: Yajun Dong
> Priority: Blocker
>
> we have a complex DAG which includes many tasks. but recently we found some tasks start delayed. for instance:
> start_task(which will start at 00:00) --> create_cluster(will finished at 00:11) --> wait_task(start at 00:16) ,
> note:
> 1. wait_task only has one upstream that is create_cluster
> 2. the server that the airflow hosts has enough memory, and celeryd_concurrency is 20.
> below is the log of wait_task:
> [2017-03-02 00:16:39,602] {models.py:124} INFO - Filling up the DagBag from /home/ubuntu/airflow/dags/etl_prod/etl_prod.py
> [2017-03-02 00:16:39,603] {models.py:197} INFO - Importing /home/ubuntu/airflow/dags/etl_prod/etl_prod.py
> [2017-03-02 00:16:39,613] {models.py:284} INFO - Loaded DAG <DAG: etl_prod>
> [2017-03-02 00:16:40,333] {models.py:124} INFO - Filling up the DagBag from /home/ubuntu/airflow/dags/etl_prod/etl_prod.py
> [2017-03-02 00:16:40,333] {models.py:197} INFO - Importing /home/ubuntu/airflow/dags/etl_prod/etl_prod.py
> [2017-03-02 00:16:40,345] {models.py:284} INFO - Loaded DAG <DAG: etl_prod>
> [2017-03-02 00:16:40,373] {models.py:936} INFO -
> --------------------------------------------------------------------------------
> New run starting @2017-03-02T00:16:40.369560
> --------------------------------------------------------------------------------
> [2017-03-02 00:16:40,402] {models.py:951} INFO - Queuing into pool None
> [2017-03-02 00:22:31,161] {models.py:124} INFO - Filling up the DagBag from /home/ubuntu/airflow/dags/etl_prod/etl_prod.py
> [2017-03-02 00:22:31,162] {models.py:197} INFO - Importing /home/ubuntu/airflow/dags/etl_prod/etl_prod.py
> [2017-03-02 00:22:31,172] {models.py:284} INFO - Loaded DAG <DAG: etl_prod>
> [2017-03-02 00:22:31,863] {models.py:124} INFO - Filling up the DagBag from /home/ubuntu/airflow/dags/etl_prod/etl_prod.py
> [2017-03-02 00:22:31,863] {models.py:197} INFO - Importing /home/ubuntu/airflow/dags/etl_prod/etl_prod.py
> [2017-03-02 00:22:31,874] {models.py:284} INFO - Loaded DAG <DAG: etl_prod>
> [2017-03-02 00:22:31,901] {models.py:936} INFO -
> --------------------------------------------------------------------------------
> New run starting @2017-03-02T00:22:31.897547
> --------------------------------------------------------------------------------
> [2017-03-02 00:22:31,911] {models.py:974} INFO - Executing <Task(BashOperator): wait_mins> on 2017-03-01 00:00:00
> [2017-03-02 00:22:31,922] {bash_operator.py:52} INFO - tmp dir root location:
> /tmp
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)