You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/06/01 17:31:33 UTC
[GitHub] [airflow] potiuk opened a new pull request, #24085: Add example for dynamically mapped tasks
potiuk opened a new pull request, #24085:
URL: https://github.com/apache/airflow/pull/24085
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragement file, named `{pr_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #24085: Add example for dynamically mapped tasks
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #24085:
URL: https://github.com/apache/airflow/pull/24085#issuecomment-1144083119
Seems like bitnami chart is failing more and more ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell commented on a diff in pull request #24085: Add example for dynamically mapped tasks
Posted by GitBox <gi...@apache.org>.
josh-fell commented on code in PR #24085:
URL: https://github.com/apache/airflow/pull/24085#discussion_r887151731
##########
airflow/example_dags/example_mapped_task.py:
##########
@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Iterable
+
+import pendulum
+
+from airflow import DAG, XComArg
+from airflow.decorators import task
+from airflow.operators.bash_operator import BashOperator
Review Comment:
```suggestion
from airflow.operators.bash import BashOperator
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on pull request #24085: Add example for dynamically mapped tasks
Posted by GitBox <gi...@apache.org>.
uranusjr commented on PR #24085:
URL: https://github.com/apache/airflow/pull/24085#issuecomment-1145679908
Looks like the _test_ is wrong. MappedOperator instances are deserialised into MappedOperator, not SerializedBaseOperator. The test should be updated to
```python
if serialized_task.is_mapped:
assert isinstance(serialized_task, MappedOperator)
assert isinstance(task, MappedOperator)
else:
assert isinstance(serialized_task, SerializedBaseOperator)
assert isinstance(task, BaseOperator)
assert not isinstance(task, SerializedBaseOperator)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #24085: Add example for dynamically mapped tasks
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #24085:
URL: https://github.com/apache/airflow/pull/24085#discussion_r887226667
##########
airflow/example_dags/example_mapped_task.py:
##########
@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Iterable
+
+import pendulum
+
+from airflow import DAG, XComArg
+from airflow.decorators import task
+from airflow.operators.bash_operator import BashOperator
Review Comment:
Aha. Right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #24085: Add example for dynamically mapped tasks
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #24085:
URL: https://github.com/apache/airflow/pull/24085#issuecomment-1145359249
Hey @uranusjr @ashb @kaxil - I need your expert knowledge here.
It seems that - after I added "Mapped tasks" to example dags, the serialization test detected it as "wrong". I am not 100% sure what the implications are without deeper looking at it but it does look like we are not able to use Mapped Tasks in cases where Dags are serialized - which might or might not be a problem.
Example here:
https://github.com/apache/airflow/runs/6710772433?check_suite_focus=true#step:10:4202
```
> assert isinstance(serialized_task, SerializedBaseOperator)
E assert False
E + where False = isinstance(<Mapped(_PythonDecoratedOperator): add_one>, SerializedBaseOperator)
```
Any comments before I dive deeper to understand it ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed pull request #24085: Add example for dynamically mapped tasks
Posted by GitBox <gi...@apache.org>.
potiuk closed pull request #24085: Add example for dynamically mapped tasks
URL: https://github.com/apache/airflow/pull/24085
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #24085: Add example for dynamically mapped tasks
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #24085:
URL: https://github.com/apache/airflow/pull/24085#issuecomment-1145982875
I'd appreciate thorough review :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #24085: Add example for dynamically mapped tasks
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #24085:
URL: https://github.com/apache/airflow/pull/24085#issuecomment-1145688810
Thanks @uranusjr !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a diff in pull request #24085: Add example for dynamically mapped tasks
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on code in PR #24085:
URL: https://github.com/apache/airflow/pull/24085#discussion_r887182765
##########
airflow/example_dags/example_mapped_task.py:
##########
@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Iterable
+
+import pendulum
+
+from airflow import DAG, XComArg
+from airflow.decorators import task
+from airflow.operators.bash_operator import BashOperator
+
+"""Example DAG demonstrating the usage of dynamically mapped tasks."""
+
+with DAG(
+ dag_id="example_mapped_task",
+ start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
+ catchup=False,
+ tags=['example'],
+) as dag:
+
+ @task
+ def make_list():
+ return [1, 2, 3]
+
+ @task
+ def add_one(x: int, multiplier: int = 1):
+ return (x + 1) * 2
Review Comment:
```suggestion
return (x + multiplier) * 2
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #24085: Add example for dynamically mapped tasks
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #24085:
URL: https://github.com/apache/airflow/pull/24085#discussion_r887226341
##########
airflow/example_dags/example_mapped_task.py:
##########
@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Iterable
+
+import pendulum
+
+from airflow import DAG, XComArg
+from airflow.decorators import task
+from airflow.operators.bash_operator import BashOperator
+
+"""Example DAG demonstrating the usage of dynamically mapped tasks."""
+
+with DAG(
+ dag_id="example_mapped_task",
+ start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
+ catchup=False,
+ tags=['example'],
+) as dag:
+
+ @task
+ def make_list():
+ return [1, 2, 3]
+
+ @task
+ def add_one(x: int, multiplier: int = 1):
+ return (x + 1) * 2
Review Comment:
Right. I focused on getting it complex and try out all the features of dymamic tasks, not to be correct :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a diff in pull request #24085: Add example for dynamically mapped tasks
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on code in PR #24085:
URL: https://github.com/apache/airflow/pull/24085#discussion_r887186056
##########
airflow/example_dags/example_mapped_task.py:
##########
@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Iterable
+
+import pendulum
+
+from airflow import DAG, XComArg
+from airflow.decorators import task
+from airflow.operators.bash_operator import BashOperator
+
+"""Example DAG demonstrating the usage of dynamically mapped tasks."""
+
+with DAG(
+ dag_id="example_mapped_task",
+ start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
+ catchup=False,
+ tags=['example'],
+) as dag:
+
+ @task
+ def make_list():
+ return [1, 2, 3]
+
+ @task
+ def add_one(x: int, multiplier: int = 1):
+ return (x + 1) * 2
Review Comment:
Good call, but shouldn't it be:
```suggestion
return (x + 1) * multiplier
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #24085: Add example for dynamically mapped tasks
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #24085:
URL: https://github.com/apache/airflow/pull/24085#issuecomment-1143930925
One of the reasons I have not found the mapped task bug in the last rc1 is that I am usually using couple of example dags to test airflow. And surprisingly I found we had no dynamic mapped task in our "example_dags" :).
Here is one I used to test rc2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #24085: Add example for dynamically mapped tasks
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #24085:
URL: https://github.com/apache/airflow/pull/24085#issuecomment-1145345054
This is strange :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #24085: Add example for dynamically mapped tasks
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #24085:
URL: https://github.com/apache/airflow/pull/24085#issuecomment-1145982469
Hey @uranusjr - actually I needed to do quite a number of other changes to make the tests pass. I am not sure however if some of those changes are how they should - be some of them were based rather on intuition than facts as the current hierarchy of operators and how it interacts with Serialization is well,, convoluted to say the least (most of it obviously due to history of adding MappedOperator on top of existing BaseOperator).
* I believe we had qute a bit of inconsistency in how we treated hash/eq of operators for BasedOperator and MappedOperator. I think I fixed it - though when I tried to move it level up (to AbstractOperator) I had some strange "MapperOperator" is not hashable even if it derived from AbstractOperator - but likely some magic of serializing/deserializing which I don't fully understand caused it.
* operators_extra_links in baseOperator were )_ rather than [] and it cause comparision with serialized version wrong (this is the one I have most reservations about - likely it shoudl be changed elsewhere
* XComArg() is recreated when serializing and since is passed as value in a dictionary, the field comparision flagged identical instances as different. I implemented proper hash there to make it hashable and usable in set comparision, and had to make a bit more complex comparision for dicts - to compare values by equality rather than identity.
* when serializing /deserializing we loose "DecoratedMappedOperator" - likely for a good reason, that's why I had to relax the check in __eq for type otherwise we would have DecoratedMappedOperator != MappedOperator. I think it might not matter that we are loosing Decorated (because I think they are equivalent after deserializing) but maybe it's worth looking at.
* I excluded some fields from comparision, because they were wildly different after serialize/deserialize. I guess it does not matter as those are only used to before mapping (serializing happens already for fully mapped task I think) but it woudl be great to double check if that is the right approach and whether we have not lost anything
For dags:
* 'partial_kwargs',
* 'operator_class',
For tasks:
* 'operator_class',
* 'partial_kwargs',
* 'mapped_op_kwargs',
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #24085: Add example for dynamically mapped tasks
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #24085:
URL: https://github.com/apache/airflow/pull/24085#issuecomment-1146075359
For some strange reason DecoratedMappedOperator is not hashable however it should (in principle) derive hash() and eq() from MappedOperator. Anyone knows what's going on here :)? Some Python woodo we do somewhere?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #24085: Add example for dynamically mapped tasks
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #24085:
URL: https://github.com/apache/airflow/pull/24085#issuecomment-1144066011
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell commented on a diff in pull request #24085: Add example for dynamically mapped tasks
Posted by GitBox <gi...@apache.org>.
josh-fell commented on code in PR #24085:
URL: https://github.com/apache/airflow/pull/24085#discussion_r887151730
##########
airflow/example_dags/example_mapped_task.py:
##########
@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Iterable
+
+import pendulum
+
+from airflow import DAG, XComArg
+from airflow.decorators import task
+from airflow.operators.bash_operator import BashOperator
Review Comment:
```suggestion
from airflow.operators.bash import BashOperator
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #24085: Add example for dynamically mapped tasks
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on PR #24085:
URL: https://github.com/apache/airflow/pull/24085#issuecomment-1145131649
Serialization error...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #24085: Add example for dynamically mapped tasks
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #24085:
URL: https://github.com/apache/airflow/pull/24085#issuecomment-1144066024
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #24085: Add example for dynamically mapped tasks
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #24085:
URL: https://github.com/apache/airflow/pull/24085#discussion_r887229837
##########
airflow/example_dags/example_mapped_task.py:
##########
@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Iterable
+
+import pendulum
+
+from airflow import DAG, XComArg
+from airflow.decorators import task
+from airflow.operators.bash import BashOperator
+
+"""Example DAG demonstrating the usage of dynamically mapped tasks."""
+
+with DAG(
+ dag_id="example_mapped_task",
+ start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
+ catchup=False,
+ tags=['example'],
+) as dag:
+
+ @task
+ def make_list():
+ return [1, 2, 3]
+
+ @task
+ def add_one(x: int, multiplier: int = 1):
+ return (x + 1) * multiplier
+
+ @task
+ def sum_it(values: Iterable[int]):
+ total = sum(values)
+ print(f"Total was {total}")
+ return total
+
+ @task
+ def sum_strings(values: Iterable[str]):
+ total = sum(int(v) for v in values)
+ print(f"Total was {total}")
+ return total
+
+ @task
+ def echo_task(value: int):
+ return f"echo {value}"
+
+ added_values = add_one.partial(multiplier=2).expand(x=make_list())
+ second = add_one.expand(x=make_list())
+ third = add_one.expand(x=make_list(), multiplier=[2, 3, 4])
+ bash_input = echo_task.expand(value=third)
+ sum_it(third)
+
+ classic = BashOperator.partial(task_id="final").expand(bash_command=bash_input)
Review Comment:
```suggestion
classic = BashOperator.partial(task_id="classic").expand(bash_command=bash_input)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org