[jira] [Commented] (OOZIE-2988) Inestigate Apache Airflow

["Daniel Becker (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OOZIE-2988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16089728#comment-16089728 ] 

Daniel Becker commented on OOZIE-2988:
--------------------------------------

I have spent some time investigating Airflow. Here are my thoughts on it:

The most important feature of Airflow is of course that the configuration files are actually (python) code. I liked this as I, too, am more comfortable with code than configuration files.

+*Workflows*+
The Airflow DAG is approximately what workflow is in Oozie. Actions (which are called tasks in Airflow) are created in the python code using _operators_, which are python object constructors. These have a set of parameters in common (they all derive from a common base class), but they also have their specific parameters. Tasks can also inherit default configuration from the DAG they are contained in.

*Remark:* I definitely think that handling default configurations is something that we should do, and Oozie workflows can already do it, for example with the <global> sections (since workflow schema 0.5).

Airflow DAGs are checked for acyclicity and exceptions are raised if the conditions are violated. When executing a workflow, Airflow evaluates the python DAG definition from time to time to check / schedule / run individual tasks (so the definition must be idempotent).

+*Transitions*+
Specifying the transitions between the actions is what I think is one of the best features of Airflow, as it is done in a simpler and more intuitive way than Oozie. After the actions are created, you can simply specify on to be upstream to another. For example you have task1 and task2, and you'd like to set task1 before task2, you can do:

{code:java}
task2.set_upstream(task1)
{code}
or
{code:java}
task1.set_downstream(task2)
{code}
or even

{code:java}
task1 >> task2
{code}

*Remark:* This way, forks and joins are handled implicitly, something we should consider.

+*Decisions*+
Decisions are handled by the BranchPythonOperator which has a python callable as a parameter. The callable returns the the id of the task that should be chosen. This task must already be downstream to the branching. When evaluating the callable, the status of all tasks downstream to the branching are set to _skipped_ with the exception of the chosen branch. There is a catch though:

{code:java}
branch_task1 -------- task2 -------- task3
                    \ ----------------/
{code}

is not correct. Here, task3 is downstream to both task1 and task2, and when evaluating the branching at branch_task1, and the path through task2 is taken, task3's status is also set to _skipped_ as it is downstream to branch_task1 and its branch has not been taken. After task2 executes, task3 will NOT be run.
The solution to this problem is to use a dummy task like this:


{code:java}
branch_task1 ------------ task2 ------------ task3
                    \ -------- dummy_task----/
{code}

Airflow has a DummyOperator class for this.
*Remarks:* I think this behaviour is unintuitive and we should not use it.

+*Dependencies of nodes*+
By default, tasks downstream to a node wait until that node finishes successfully, and if it fails, they are not started. This can be customised.

*Remarks:* Oozie already has its model of action dependency (with _ok_ and _error_ transitions), I think we should use the existing practice.

+*Coordinators and bundles*+
Scheduling is an integrated part of DAGs, they are not handled separately from the workflow definition.
*Remark:* It could be a good decision, but in Oozie we already handle workflows and coordinators separately (which is also a practice we can argue for), and I think it would be too big a change to do it differently in our API.

Bundle are not present in Airflow either, but we can use nested sub-DAGs to treat one or more workflows as a unit in a larger workflows. Oozie supports subworkflows too.

+*Other things*+
Airflow has support for packaging dependencies together with the DAGs, although these dependencies have to be written in python.
Airflow supports Jinja templates: [http://jinja.pocoo.org/docs/dev/].

> Inestigate Apache Airflow
> -------------------------
>
>                 Key: OOZIE-2988
>                 URL: https://issues.apache.org/jira/browse/OOZIE-2988
>             Project: Oozie
>          Issue Type: Sub-task
>          Components: client
>            Reporter: Andras Piros
>            Assignee: Daniel Becker
>   Original Estimate: 16h
>          Time Spent: 14h
>  Remaining Estimate: 2h
>
> Investigate current version of [*Apache Airflow*|https://airflow.incubator.apache.org/concepts.html] regarding following:
> * how to define a DAG
> ** nodes
> ** edges
> ** transitions
> ** decision and fork / join nodes
> ** checks on DAGness, and on completeness
> * how are connections between DAGs handled
> ** state of one running / already finished DAG is respected by another DAG
> * creation / running process
> ** what to define w/ Python code
> ** and what using templates
> ** inheritance in Python
> ** third party libs in Python code while creating a DAG
> ** how are DAGs stored and submitted / run
> ** what patterns are used under the hood (builder, delegate, ...)
> ** how is logic considered (Oozie EL functions)
> * how to schedule a DAG (Oozie Coordinators)
> * how to bundle these together (Oozie Bundles)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)