You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "uranusjr (via GitHub)" <gi...@apache.org> on 2024/04/01 06:40:38 UTC

[PR] Implement Metadata to emit runtime extra [airflow]

uranusjr opened a new pull request, #38650:
URL: https://github.com/apache/airflow/pull/38650

   Close #37810. This implements the `yield` syntax and the `Metadata` class, and mechanism to collect the yielded values in the worker.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Implement Metadata to emit runtime extra [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr merged PR #38650:
URL: https://github.com/apache/airflow/pull/38650


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Implement Metadata to emit runtime extra [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #38650:
URL: https://github.com/apache/airflow/pull/38650#discussion_r1548815506


##########
airflow/operators/python.py:
##########
@@ -232,7 +232,11 @@ def execute(self, context: Context) -> Any:
         context_merge(context, self.op_kwargs, templates_dict=self.templates_dict)
         self.op_kwargs = self.determine_kwargs(context)
 
-        return_value = self.execute_callable()
+        try:
+            dataset_events = context["dataset_events"]
+        except KeyError:
+            dataset_events = context["dataset_events"] = DatasetEventAccessors()
+        return_value = self.execute_callable(dataset_events)

Review Comment:
   Because `execute_callable` does not receive the context at all. If you ask me it’s sort of designed quite weirdly to begin with.
   
   I think I have a way to avoid not passing in the additional argument, by storing the value on the operator (`self`) instead. It’s a bit hacky and more of a workaround.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Implement Metadata to emit runtime extra [airflow]

Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on code in PR #38650:
URL: https://github.com/apache/airflow/pull/38650#discussion_r1546108388


##########
airflow/datasets/metadata.py:
##########
@@ -0,0 +1,39 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any
+
+import attrs
+
+from airflow.datasets import coerce_to_uri
+
+if TYPE_CHECKING:
+    from airflow.datasets import Dataset
+
+
+@attrs.define(init=False)
+class Metadata:

Review Comment:
   Okay, took a while to find it... nevermind.



##########
airflow/datasets/metadata.py:
##########
@@ -0,0 +1,39 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any
+
+import attrs
+
+from airflow.datasets import coerce_to_uri
+
+if TYPE_CHECKING:
+    from airflow.datasets import Dataset
+
+
+@attrs.define(init=False)
+class Metadata:

Review Comment:
   Okay, took a while to find it... nevermind.



##########
airflow/datasets/metadata.py:
##########
@@ -0,0 +1,39 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any
+
+import attrs
+
+from airflow.datasets import coerce_to_uri
+
+if TYPE_CHECKING:
+    from airflow.datasets import Dataset
+
+
+@attrs.define(init=False)
+class Metadata:

Review Comment:
   Where in this PR is the new `Metadata` class actually being used? Was looking for it but could not see a reference. Not that it is "fresh dead code"?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Implement Metadata to emit runtime extra [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #38650:
URL: https://github.com/apache/airflow/pull/38650#discussion_r1546064643


##########
docs/apache-airflow/authoring-and-scheduling/datasets.rst:
##########
@@ -232,20 +232,36 @@ Attaching extra information to an emitting Dataset Event
 A task with a dataset outlet can optionally attach extra information before it emits a dataset event. This is different
 from `Extra information on Dataset`_. Extra information on a dataset statically describes the entity pointed to by the dataset URI; extra information on the *dataset event* instead should be used to annotate the triggering data change, such as how many rows in the database are changed by the update, or the date range covered by it.
 
-The easiest way to attach extra information to the dataset event is by accessing ``dataset_events`` in a task's execution context:
+The easiest way to attach extra information to the dataset event is by ``yield``-ing a ``Metadata`` object from a task:
 
 .. code-block:: python
 
+    from airflow.datasets import Dataset
+    from airflow.datasets.metadata import Metadata
+
     example_s3_dataset = Dataset("s3://dataset/example.csv")
 
 
     @task(outlets=[example_s3_dataset])
-    def write_to_s3(*, dataset_events):
+    def write_to_s3():
         df = ...  # Get a Pandas DataFrame to write.
         # Write df to dataset...
+        yield Metadata(example_s3_dataset, {"row_count": len(df)})
+
+Airflow automatically collects all yielded metadata, and populates dataset events with extra information for corresponding metadata objects.
+
+This can also be done in classic operators. The best way is to subclass the operator and override ``execute``. Alternatively, extras can also be added in a task's ``pre_execute`` or ``post_execute`` hook.

Review Comment:
   I added a note about this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Implement Metadata to emit runtime extra [airflow]

Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on code in PR #38650:
URL: https://github.com/apache/airflow/pull/38650#discussion_r1546089068


##########
airflow/operators/python.py:
##########
@@ -232,7 +232,11 @@ def execute(self, context: Context) -> Any:
         context_merge(context, self.op_kwargs, templates_dict=self.templates_dict)
         self.op_kwargs = self.determine_kwargs(context)
 
-        return_value = self.execute_callable()
+        try:
+            dataset_events = context["dataset_events"]
+        except KeyError:
+            dataset_events = context["dataset_events"] = DatasetEventAccessors()
+        return_value = self.execute_callable(dataset_events)

Review Comment:
   I am not sure why we need that change to add `dataset_events` to the calling args - we don't add other context information here as well. Why should be dataset event be different than other context vars?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Implement Metadata to emit runtime extra [airflow]

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #38650:
URL: https://github.com/apache/airflow/pull/38650#discussion_r1546019417


##########
docs/apache-airflow/authoring-and-scheduling/datasets.rst:
##########
@@ -232,20 +232,36 @@ Attaching extra information to an emitting Dataset Event
 A task with a dataset outlet can optionally attach extra information before it emits a dataset event. This is different
 from `Extra information on Dataset`_. Extra information on a dataset statically describes the entity pointed to by the dataset URI; extra information on the *dataset event* instead should be used to annotate the triggering data change, such as how many rows in the database are changed by the update, or the date range covered by it.
 
-The easiest way to attach extra information to the dataset event is by accessing ``dataset_events`` in a task's execution context:
+The easiest way to attach extra information to the dataset event is by ``yield``-ing a ``Metadata`` object from a task:
 
 .. code-block:: python
 
+    from airflow.datasets import Dataset
+    from airflow.datasets.metadata import Metadata
+
     example_s3_dataset = Dataset("s3://dataset/example.csv")
 
 
     @task(outlets=[example_s3_dataset])
-    def write_to_s3(*, dataset_events):
+    def write_to_s3():
         df = ...  # Get a Pandas DataFrame to write.
         # Write df to dataset...
+        yield Metadata(example_s3_dataset, {"row_count": len(df)})
+
+Airflow automatically collects all yielded metadata, and populates dataset events with extra information for corresponding metadata objects.
+
+This can also be done in classic operators. The best way is to subclass the operator and override ``execute``. Alternatively, extras can also be added in a task's ``pre_execute`` or ``post_execute`` hook.

Review Comment:
   pre and post don't have retry mechanism so I assume there can be cases where not all data collected. So there are pros and cons if users decided to do it in execute() or in pre/post
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Implement Metadata to emit runtime extra [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #38650:
URL: https://github.com/apache/airflow/pull/38650#discussion_r1546017906


##########
airflow/operators/python.py:
##########
@@ -243,13 +243,14 @@ def execute(self, context: Context) -> Any:
     def determine_kwargs(self, context: Mapping[str, Any]) -> Mapping[str, Any]:
         return KeywordParameters.determine(self.python_callable, self.op_args, context).unpacking()
 
-    def execute_callable(self) -> Any:
+    def execute_callable(self, dataset_events: DatasetEventAccessors) -> Any:

Review Comment:
   I had to change the signature here (and in subclasses) so the callable can access the accessors. I _think_ this shouldn’t be public and thus not backward incompatible? Not exactly sure…



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Implement Metadata to emit runtime extra [airflow]

Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on code in PR #38650:
URL: https://github.com/apache/airflow/pull/38650#discussion_r1546075083


##########
airflow/operators/python.py:
##########
@@ -243,13 +243,14 @@ def execute(self, context: Context) -> Any:
     def determine_kwargs(self, context: Mapping[str, Any]) -> Mapping[str, Any]:
         return KeywordParameters.determine(self.python_callable, self.op_args, context).unpacking()
 
-    def execute_callable(self) -> Any:
+    def execute_callable(self, dataset_events: DatasetEventAccessors) -> Any:

Review Comment:
   Mhm, it is not exactly defined as "public" but at least we have also a custom subclass of python operator developed in our environment. So everybody having a subclass of this will see a breaking change?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Implement Metadata to emit runtime extra [airflow]

Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on code in PR #38650:
URL: https://github.com/apache/airflow/pull/38650#discussion_r1546104066


##########
airflow/datasets/metadata.py:
##########
@@ -0,0 +1,39 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any
+
+import attrs
+
+from airflow.datasets import coerce_to_uri
+
+if TYPE_CHECKING:
+    from airflow.datasets import Dataset
+
+
+@attrs.define(init=False)
+class Metadata:

Review Comment:
   Where in this PR is the new `Metadata` class actually being used? Was looking for it but could not see a reference. Noth that it is "fresh dead code"?



##########
airflow/datasets/metadata.py:
##########
@@ -0,0 +1,39 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Any
+
+import attrs
+
+from airflow.datasets import coerce_to_uri
+
+if TYPE_CHECKING:
+    from airflow.datasets import Dataset
+
+
+@attrs.define(init=False)
+class Metadata:

Review Comment:
   Where in this PR is the new `Metadata` class actually being used? Was looking for it but could not see a reference. Not that it is "fresh dead code"?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org