You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Kaxil Naik (JIRA)" <ji...@apache.org> on 2018/09/29 14:21:00 UTC
[jira] [Comment Edited] (AIRFLOW-3118) DAGs not successful on new
installation
[ https://issues.apache.org/jira/browse/AIRFLOW-3118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16633028#comment-16633028 ]
Kaxil Naik edited comment on AIRFLOW-3118 at 9/29/18 2:20 PM:
--------------------------------------------------------------
[~ashb] I dug into it. Looks like this bug has been since the inception of Airflow. I tested it with 1.8.2, 1.9, 1.10 and the latest master. I think it will take more time than I expected. Won't be working on this for a while, preparing for my 3-hour long presentation on Airflow in my company :) Feel free to work on this.
The solution should be to change https://github.com/apache/incubator-airflow/blob/7a6f4b013335d42eb7005f49ff202da147b90af1/airflow/jobs.py#L1688-L1711
{noformat}
simple_dag_bag = SimpleDagBag(simple_dags)
if len(simple_dags) > 0:
# Handle cases where a DAG run state is set (perhaps manually) to
# a non-running state. Handle task instances that belong to
# DAG runs in those states
# If a task instance is up for retry but the corresponding DAG run
# isn't running, mark the task instance as FAILED so we don't try
# to re-run it.
self._change_state_for_tis_without_dagrun(simple_dag_bag,
[State.UP_FOR_RETRY],
State.FAILED)
# If a task instance is scheduled or queued, but the corresponding
# DAG run isn't running, set the state to NONE so we don't try to
# re-run it.
self._change_state_for_tis_without_dagrun(simple_dag_bag,
[State.QUEUED,
State.SCHEDULED],
State.NONE)
self._execute_task_instances(simple_dag_bag,
(State.SCHEDULED,))
{noformat}
was (Author: kaxilnaik):
[~ashb] I dug into it. Looks like this bug has been since the inception of Airflow. I tested it with 1.8.2, 1.9, 1.10 and the latest master. I think it will take more time than I expected. Won't be working on this for a while, preparing for my 3-hour long presentation on Airflow in my company :)
The solution should be to change https://github.com/apache/incubator-airflow/blob/7a6f4b013335d42eb7005f49ff202da147b90af1/airflow/jobs.py#L1688-L1711
{noformat}
simple_dag_bag = SimpleDagBag(simple_dags)
if len(simple_dags) > 0:
# Handle cases where a DAG run state is set (perhaps manually) to
# a non-running state. Handle task instances that belong to
# DAG runs in those states
# If a task instance is up for retry but the corresponding DAG run
# isn't running, mark the task instance as FAILED so we don't try
# to re-run it.
self._change_state_for_tis_without_dagrun(simple_dag_bag,
[State.UP_FOR_RETRY],
State.FAILED)
# If a task instance is scheduled or queued, but the corresponding
# DAG run isn't running, set the state to NONE so we don't try to
# re-run it.
self._change_state_for_tis_without_dagrun(simple_dag_bag,
[State.QUEUED,
State.SCHEDULED],
State.NONE)
self._execute_task_instances(simple_dag_bag,
(State.SCHEDULED,))
{noformat}
> DAGs not successful on new installation
> ---------------------------------------
>
> Key: AIRFLOW-3118
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3118
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG
> Affects Versions: 1.10.0
> Environment: Ubuntu 18.04
> Python 3.6
> Reporter: Brylie Christopher Oxley
> Priority: Blocker
> Attachments: Screenshot_20180926_161837.png, image-2018-09-26-12-39-03-094.png
>
>
> When trying out Airflow, on localhost, none of the DAG runs are getting to the 'success' state. They are getting stuck in 'running', or I manually label them as failed:
> !image-2018-09-26-12-39-03-094.png!
> h2. Steps to reproduce
> # create new conda environment
> ** conda create -n airflow
> ** source activate airflow
> # install airflow
> ** pip install apache-airflow
> # initialize Airflow db
> ** airflow initdb
> # disable default paused setting in airflow.cfg
> ** dags_are_paused_at_creation = False
> # {color:#6a8759}run airflow and airflow scheduler (in separate terminal){color}
> ** {color:#6a8759}airflow scheduler{color}
> ** {color:#6a8759}airflow webserver{color}
> # {color:#6a8759}unpause example_bash_operator{color}
> ** {color:#6a8759}airflow unpause example_bash_operator{color}
> # {color:#6a8759}log in to Airflow UI{color}
> # {color:#6a8759}turn on example_bash_operator{color}
> # {color:#6a8759}click "Trigger DAG" in `example_bash_operator` row{color}
> h2. {color:#6a8759}Observed result{color}
> {color:#6a8759}The `example_bash_operator` never leaves the "running" state.{color}
> h2. {color:#6a8759}Expected result{color}
> {color:#6a8759}The `example_bash_operator` would quickly enter the "success" state{color}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)