You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/02/10 22:12:13 UTC
[GitHub] [airflow] uranusjr opened a new pull request #21505: MappedOperator redesign
uranusjr opened a new pull request #21505:
URL: https://github.com/apache/airflow/pull/21505
This PR may be too large.
Details to come later. Submitted for CI at the moment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr edited a comment on pull request #21505: MappedOperator redesign
Posted by GitBox <gi...@apache.org>.
uranusjr edited a comment on pull request #21505:
URL: https://github.com/apache/airflow/pull/21505#issuecomment-1036292418
Alright, this is passing. I’ve updated the top description to add details about the changeset. I’m not quite sure how I can split the PR into multiple parts, so let’s try doing everything in one shot first 😢
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #21505: MappedOperator redesign
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #21505:
URL: https://github.com/apache/airflow/pull/21505#discussion_r804800178
##########
File path: airflow/models/taskmixin.py
##########
@@ -309,4 +311,4 @@ def has_mapped_dependants(self) -> bool:
only contains operators, not task groups. In the future, we should
provide a way to record an DAG node's all downstream nodes instead.
"""
- return any(self.mapped_dependants())
+ return next(self.mapped_dependants(), None) is not None
Review comment:
Why this change? `any()` seems clearer to me than calling `next()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #21505: MappedOperator redesign
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #21505:
URL: https://github.com/apache/airflow/pull/21505#discussion_r804822924
##########
File path: airflow/models/taskmixin.py
##########
@@ -309,4 +311,4 @@ def has_mapped_dependants(self) -> bool:
only contains operators, not task groups. In the future, we should
provide a way to record an DAG node's all downstream nodes instead.
"""
- return any(self.mapped_dependants())
+ return next(self.mapped_dependants(), None) is not None
Review comment:
It’s _marginally_ faster. I’ll revert.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #21505: MappedOperator redesign
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #21505:
URL: https://github.com/apache/airflow/pull/21505#discussion_r804770795
##########
File path: airflow/models/abstractoperator.py
##########
@@ -0,0 +1,247 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import datetime
+from typing import TYPE_CHECKING, Any, Callable, Collection, Dict, List, Optional, Set, Type, Union
+
+import jinja2
Review comment:
Jinja is slow to import -- so we should move this import, and the `airflow.templates` to in to the functions/`if TYPE_CHECKING` blocks please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #21505: MappedOperator redesign
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #21505:
URL: https://github.com/apache/airflow/pull/21505#discussion_r804822315
##########
File path: airflow/models/taskinstance.py
##########
@@ -1840,7 +1851,7 @@ def _get_previous_dagrun_success() -> Optional["DagRun"]:
def _get_previous_dagrun_data_interval_success() -> Optional["DataInterval"]:
dagrun = _get_previous_dagrun_success()
- if dagrun is None:
+ if dagrun is None or dag is None:
Review comment:
Because local functions capture `dag` lazily and Mypy is hard to please :(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #21505: MappedOperator redesign
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #21505:
URL: https://github.com/apache/airflow/pull/21505#discussion_r804826652
##########
File path: airflow/decorators/base.py
##########
@@ -337,30 +361,47 @@ def _merge_kwargs(
return {**kwargs1, **kwargs2}
-@attr.define(kw_only=True)
+@attr.define(kw_only=True, repr=False)
class DecoratedMappedOperator(MappedOperator):
"""MappedOperator implementation for @task-decorated task function."""
multiple_outputs: bool
python_callable: Callable
- def create_unmapped_operator(self, dag: "DAG") -> BaseOperator:
+ # We can't save these in partial_kwargs because op_args and op_kwargs need
+ # to be present in mapped_kwargs, and MappedOperator prevents duplication.
+ partial_op_kwargs: Dict[str, Any]
+
+ @classmethod
+ @cache
+ def get_serialized_fields(cls):
+ return MappedOperator.get_serialized_fields() | {"partial_op_kwargs"}
Review comment:
Hmm no this is due to `@cache`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on pull request #21505: MappedOperator redesign
Posted by GitBox <gi...@apache.org>.
uranusjr commented on pull request #21505:
URL: https://github.com/apache/airflow/pull/21505#issuecomment-1036292418
Alright, this is passing. I’ve updated the top description to add details about the changeset. I’m not quite sure how I can split the PR into multiple parts 😢
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #21505: MappedOperator redesign
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #21505:
URL: https://github.com/apache/airflow/pull/21505#discussion_r804852148
##########
File path: airflow/models/taskinstance.py
##########
@@ -1840,7 +1851,7 @@ def _get_previous_dagrun_success() -> Optional["DagRun"]:
def _get_previous_dagrun_data_interval_success() -> Optional["DataInterval"]:
dagrun = _get_previous_dagrun_success()
- if dagrun is None:
+ if dagrun is None or dag is None:
Review comment:
Slightly changed assert to avoid this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #21505: MappedOperator redesign
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #21505:
URL: https://github.com/apache/airflow/pull/21505#discussion_r804798750
##########
File path: airflow/models/taskinstance.py
##########
@@ -1812,8 +1821,10 @@ def get_template_context(
integrate_macros_plugins()
- task: "BaseOperator" = self.task
- dag: DAG = task.dag
+ task = self.task
+ dag: Optional[DAG] = task.dag
+ assert dag is not None # For Mypy.
Review comment:
```suggestion
assert dag # For Mypy.
```
For consistency
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #21505: MappedOperator redesign
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #21505:
URL: https://github.com/apache/airflow/pull/21505#discussion_r804799566
##########
File path: airflow/models/taskinstance.py
##########
@@ -1840,7 +1851,7 @@ def _get_previous_dagrun_success() -> Optional["DagRun"]:
def _get_previous_dagrun_data_interval_success() -> Optional["DataInterval"]:
dagrun = _get_previous_dagrun_success()
- if dagrun is None:
+ if dagrun is None or dag is None:
Review comment:
Haven't you already `assert dag is not None?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #21505: MappedOperator redesign
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #21505:
URL: https://github.com/apache/airflow/pull/21505#discussion_r804872002
##########
File path: airflow/decorators/base.py
##########
@@ -337,30 +361,47 @@ def _merge_kwargs(
return {**kwargs1, **kwargs2}
-@attr.define(kw_only=True)
+@attr.define(kw_only=True, repr=False)
class DecoratedMappedOperator(MappedOperator):
"""MappedOperator implementation for @task-decorated task function."""
multiple_outputs: bool
python_callable: Callable
- def create_unmapped_operator(self, dag: "DAG") -> BaseOperator:
+ # We can't save these in partial_kwargs because op_args and op_kwargs need
+ # to be present in mapped_kwargs, and MappedOperator prevents duplication.
+ partial_op_kwargs: Dict[str, Any]
+
+ @classmethod
+ @cache
+ def get_serialized_fields(cls):
+ return MappedOperator.get_serialized_fields() | {"partial_op_kwargs"}
Review comment:
Using the “old style” `super` to work around this.
```python
sup = super(DecoratedMappedOperator, DecoratedMappedOperator)
return sup.get_serialized_fields() | {"partial_op_kwargs"}
```
Experience with Python 2 is still useful sometimes 🙂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #21505: MappedOperator redesign
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #21505:
URL: https://github.com/apache/airflow/pull/21505#discussion_r804767584
##########
File path: airflow/decorators/base.py
##########
@@ -337,30 +361,47 @@ def _merge_kwargs(
return {**kwargs1, **kwargs2}
-@attr.define(kw_only=True)
+@attr.define(kw_only=True, repr=False)
class DecoratedMappedOperator(MappedOperator):
"""MappedOperator implementation for @task-decorated task function."""
multiple_outputs: bool
python_callable: Callable
- def create_unmapped_operator(self, dag: "DAG") -> BaseOperator:
+ # We can't save these in partial_kwargs because op_args and op_kwargs need
+ # to be present in mapped_kwargs, and MappedOperator prevents duplication.
+ partial_op_kwargs: Dict[str, Any]
+
+ @classmethod
+ @cache
+ def get_serialized_fields(cls):
+ return MappedOperator.get_serialized_fields() | {"partial_op_kwargs"}
Review comment:
Any reason to not user super here?
```suggestion
return super().get_serialized_fields() | {"partial_op_kwargs"}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr merged pull request #21505: Straighten up MappedOperator hierarchy and typing
Posted by GitBox <gi...@apache.org>.
uranusjr merged pull request #21505:
URL: https://github.com/apache/airflow/pull/21505
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #21505: MappedOperator redesign
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #21505:
URL: https://github.com/apache/airflow/pull/21505#discussion_r804793596
##########
File path: airflow/decorators/base.py
##########
@@ -337,30 +361,47 @@ def _merge_kwargs(
return {**kwargs1, **kwargs2}
-@attr.define(kw_only=True)
+@attr.define(kw_only=True, repr=False)
class DecoratedMappedOperator(MappedOperator):
"""MappedOperator implementation for @task-decorated task function."""
multiple_outputs: bool
python_callable: Callable
- def create_unmapped_operator(self, dag: "DAG") -> BaseOperator:
+ # We can't save these in partial_kwargs because op_args and op_kwargs need
+ # to be present in mapped_kwargs, and MappedOperator prevents duplication.
+ partial_op_kwargs: Dict[str, Any]
+
+ @classmethod
+ @cache
+ def get_serialized_fields(cls):
+ return MappedOperator.get_serialized_fields() | {"partial_op_kwargs"}
Review comment:
Somehow `super()` doesn’t work here:
```pycon
>>> from airflow.decorators.base import DecoratedMappedOperator
>>> DecoratedMappedOperator.get_serialized_fields()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/airflow/airflow/decorators/base.py", line 378, in get_serialized_fields
return super().get_serialized_fields() | {"partial_op_kwargs"}
TypeError: super(type, obj): obj must be an instance or subtype of type
```
I’m suspecting this has something to do with `@attr.define`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #21505: MappedOperator redesign
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #21505:
URL: https://github.com/apache/airflow/pull/21505#issuecomment-1036381617
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org