You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/10/13 12:03:18 UTC
[GitHub] [airflow] ashb opened a new pull request #11497: Rename "functional DAGs" to "Decorated Flows"
ashb opened a new pull request #11497:
URL: https://github.com/apache/airflow/pull/11497
Functional DAGs were so called because the DAG is "made up of funcitons"
but this AIP adds much more than just the task decorator change -- it
adds nicer XCom use, and in many cases automatic dependencies between
tasks.
"Functional" also invokes "functional programming" which this isn't.
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #11497: Rename "functional DAGs" to "Decorated Flows"
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #11497:
URL: https://github.com/apache/airflow/pull/11497#discussion_r503912045
##########
File path: docs/concepts.rst
##########
@@ -116,17 +116,21 @@ DAGs can be used as context managers to automatically assign new operators to th
op.dag is dag # True
-.. _concepts:functional_dags:
+.. _concepts:decorated_flows:
-Functional DAGs
+Decorated Flows
---------------
-DAGs can be defined using functional abstractions. Outputs and inputs are sent between tasks using
-:ref:`XCom values <concepts:xcom>`. In addition, you can wrap functions as tasks using the
-:ref:`task decorator <concepts:task_decorator>`. Airflow will also automatically add dependencies between
-tasks to ensure that XCom messages are available when operators are executed.
+.. versionadded:: 2.0.0
-Example DAG with functional abstraction
+Airflow 2.0 adds a new style of authoring dags called Decorated Flows which removes a lot of the boilerplate
Review comment:
Should we add something like `(aka functional DAGs)` just to point people who already know about it that this is the same thing?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #11497: Rename "functional DAGs" to "Decorated Flows"
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #11497:
URL: https://github.com/apache/airflow/pull/11497#issuecomment-707908029
this aip is about so much more than just the task decorator. The bigger thing is the change to xcom and dependency management!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] casassg commented on pull request #11497: Rename "functional DAGs" to "Decorated Flows"
Posted by GitBox <gi...@apache.org>.
casassg commented on pull request #11497:
URL: https://github.com/apache/airflow/pull/11497#issuecomment-707856537
I'm not sure about decorated flows. I understand why not Functional DAGs but decorated flows seems a bit of a weird naming?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #11497: Rename "functional DAGs" to "Decorated Flows"
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #11497:
URL: https://github.com/apache/airflow/pull/11497#issuecomment-707727807
I'll wait for docs to build then merge this
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #11497: Rename "functional DAGs" to "Decorated Flows"
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #11497:
URL: https://github.com/apache/airflow/pull/11497#discussion_r503912583
##########
File path: docs/concepts.rst
##########
@@ -116,17 +116,21 @@ DAGs can be used as context managers to automatically assign new operators to th
op.dag is dag # True
-.. _concepts:functional_dags:
+.. _concepts:decorated_flows:
-Functional DAGs
+Decorated Flows
---------------
-DAGs can be defined using functional abstractions. Outputs and inputs are sent between tasks using
-:ref:`XCom values <concepts:xcom>`. In addition, you can wrap functions as tasks using the
-:ref:`task decorator <concepts:task_decorator>`. Airflow will also automatically add dependencies between
-tasks to ensure that XCom messages are available when operators are executed.
+.. versionadded:: 2.0.0
-Example DAG with functional abstraction
+Airflow 2.0 adds a new style of authoring dags called Decorated Flows which removes a lot of the boilerplate
Review comment:
Good idea
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #11497: Rename "functional DAGs" to "Decorated Flows"
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #11497:
URL: https://github.com/apache/airflow/pull/11497#issuecomment-707699562
cc @casassg We've renamed your functional DAGs to "Decorated Flows". (We can still change this pre 2.0.0beta if anyone disagrees and comes up with a better name)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb merged pull request #11497: Rename "functional DAGs" to "Decorated Flows"
Posted by GitBox <gi...@apache.org>.
ashb merged pull request #11497:
URL: https://github.com/apache/airflow/pull/11497
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman commented on pull request #11497: Rename "functional DAGs" to "Decorated Flows"
Posted by GitBox <gi...@apache.org>.
dimberman commented on pull request #11497:
URL: https://github.com/apache/airflow/pull/11497#issuecomment-707893038
I'm not a huge fan either, how about `Task Functions` or `Task Wrappers`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] casassg commented on pull request #11497: Rename "functional DAGs" to "Decorated Flows"
Posted by GitBox <gi...@apache.org>.
casassg commented on pull request #11497:
URL: https://github.com/apache/airflow/pull/11497#issuecomment-708096396
Agree. But decorated flows seems weird still. Also, note that even though it's not unlocking functional programming entirely it does focus on functions as the main building block.
How about Pythonic DAGs? Given most of the improvements are aimed at making the DAG writing process look more like how you would normally write python (function, arguments, implicit dependency). Not sure still, open to opinions. Also should we bring this to Slack or dev list?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #11497: Rename "functional DAGs" to "Decorated Flows"
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #11497:
URL: https://github.com/apache/airflow/pull/11497#issuecomment-707782106
I've run `breeze build-docs` locally and it passes fine.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman commented on pull request #11497: Rename "functional DAGs" to "Decorated Flows"
Posted by GitBox <gi...@apache.org>.
dimberman commented on pull request #11497:
URL: https://github.com/apache/airflow/pull/11497#issuecomment-707898373
@potiuk @casassg @kaxil @ashb WDYT about `task function`? I agree that `functional` didn't make sense since this is not FP, but what we're essentially is taking native python functions and turning them into tasks.... so... task functions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org