You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jedcunningham (via GitHub)" <gi...@apache.org> on 2023/02/27 23:36:26 UTC
[GitHub] [airflow] jedcunningham opened a new pull request, #29793: Support setup and teardown taskflow taskgroups
jedcunningham opened a new pull request, #29793:
URL: https://github.com/apache/airflow/pull/29793
This adds support for marking whole taskflow taskgroups as "setup" or "teardown" in a DAG. This provides DAG authors more flexibility, and allows easier reuse of the existing Airflow ecosystem (e.g. not everything becomes hook invocations in a single task).
This also adds some validation to make sure setup and teardown taskflow task groups can't themselves contain explicity marked setup or teardown tasks. All tasks in a setup or teardown taskflow task group are already implicitly either setup or teardown.
Support for classic style taskgroups will be handled later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on a diff in pull request #29793: Support setup and teardown taskflow taskgroups
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on code in PR #29793:
URL: https://github.com/apache/airflow/pull/29793#discussion_r1119477350
##########
tests/serialization/test_dag_serialization.py:
##########
@@ -1366,6 +1366,88 @@ def _check_taskgroup_children(
["group1.group2.teardown2"],
)
+ def test_task_group_setup_teardown_taskgroups(self):
+ """
+ Test TaskGroup setup and teardown taskgroup serialization/deserialization.
+ """
+ from airflow.decorators import setup, task_group, teardown
+ from airflow.operators.empty import EmptyOperator
+
+ execution_date = datetime(2020, 1, 1)
+ with DAG("test_task_group_setup_teardown_task_groups", start_date=execution_date) as dag:
+
+ @setup
+ @task_group
+ def setup():
+ @task_group
+ def sub_setup():
+ EmptyOperator(task_id="setup2")
+
+ EmptyOperator(task_id="setup1")
+ sub_setup()
+
+ @teardown
+ @task_group
+ def teardown():
+ EmptyOperator(task_id="teardown1")
+
+ setup()
+ EmptyOperator(task_id="sometask")
+ teardown()
+
+ dag_dict = SerializedDAG.to_dict(dag)
+ SerializedDAG.validate_schema(dag_dict)
+ json_dag = SerializedDAG.from_json(SerializedDAG.to_json(dag))
+ self.validate_deserialized_dag(json_dag, dag)
+
+ serialized_dag = SerializedDAG.deserialize_dag(SerializedDAG.serialize_dag(dag))
+
+ def _check_taskgroup_children(
+ se_task_group, dag_task_group, expected_setup, expected_children, expected_teardown
+ ):
+ assert list(se_task_group.children.keys()) == expected_children
+ assert list(dag_task_group.children.keys()) == expected_children
+
+ assert list(se_task_group.setup_children.keys()) == expected_setup
+ assert list(dag_task_group.setup_children.keys()) == expected_setup
+
+ assert list(se_task_group.teardown_children.keys()) == expected_teardown
+ assert list(dag_task_group.teardown_children.keys()) == expected_teardown
Review Comment:
```suggestion
for group in (se_task_group, dag_task_group):
assert list(group.children.keys()) == expected_children
assert list(group.setup_children.keys()) == expected_setup
assert list(group.teardown_children.keys()) == expected_teardown
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a diff in pull request #29793: Support setup and teardown taskflow taskgroups
Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham commented on code in PR #29793:
URL: https://github.com/apache/airflow/pull/29793#discussion_r1120277966
##########
tests/serialization/test_dag_serialization.py:
##########
@@ -1366,6 +1366,88 @@ def _check_taskgroup_children(
["group1.group2.teardown2"],
)
+ def test_task_group_setup_teardown_taskgroups(self):
+ """
+ Test TaskGroup setup and teardown taskgroup serialization/deserialization.
+ """
+ from airflow.decorators import setup, task_group, teardown
+ from airflow.operators.empty import EmptyOperator
+
+ execution_date = datetime(2020, 1, 1)
+ with DAG("test_task_group_setup_teardown_task_groups", start_date=execution_date) as dag:
+
+ @setup
+ @task_group
+ def setup():
Review Comment:
Oops 🤦♂️. Not the last time I'll do that most likely. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on a diff in pull request #29793: Support setup and teardown taskflow taskgroups
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on code in PR #29793:
URL: https://github.com/apache/airflow/pull/29793#discussion_r1119478850
##########
tests/serialization/test_dag_serialization.py:
##########
@@ -1366,6 +1366,88 @@ def _check_taskgroup_children(
["group1.group2.teardown2"],
)
+ def test_task_group_setup_teardown_taskgroups(self):
+ """
+ Test TaskGroup setup and teardown taskgroup serialization/deserialization.
+ """
+ from airflow.decorators import setup, task_group, teardown
+ from airflow.operators.empty import EmptyOperator
+
+ execution_date = datetime(2020, 1, 1)
+ with DAG("test_task_group_setup_teardown_task_groups", start_date=execution_date) as dag:
+
+ @setup
+ @task_group
+ def setup():
Review Comment:
probably not a good idea to redefine setup here! this bit us before :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham merged pull request #29793: Support setup and teardown taskflow taskgroups
Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham merged PR #29793:
URL: https://github.com/apache/airflow/pull/29793
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org