You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "badrisrinivas9 (via GitHub)" <gi...@apache.org> on 2023/03/10 11:49:08 UTC
[GitHub] [airflow] badrisrinivas9 opened a new issue, #30021: Nested Taskgroup and Dynamic task mapping issue
badrisrinivas9 opened a new issue, #30021:
URL: https://github.com/apache/airflow/issues/30021
### Apache Airflow version
2.5.1
### What happened
If we create two nested groups and expand the groups the second group tasks fail. I assume the second expand doesn't know which value of mapped index to take.
### What you think should happen instead
The second group should have got the value from the upstream from the first group(most probably by mapping value)
### How to reproduce
import pendulum
from airflow.decorators import dag, task, task_group
from airflow.operators.empty import EmptyOperator
@dag(dag_id="test", start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
schedule=None, catchup=False,
render_template_as_native_obj=True
)
def generic_wrapper():
task1 = EmptyOperator(task_id="task1")
task2 = EmptyOperator(task_id="task2")
@task
def task3():
return [1, 2, 3]
@task_group(
group_id="group2"
)
def group2(i):
@task
def task4(i):
print(i)
@task
def task5():
return [1, 2, 3]
task4(i) >> group3.expand(x=task5())
@task_group(
group_id="group3"
)
def group3(x):
@task
def task6(x):
print(x)
task6(x)
task1 >> task2 >> group2.expand(i=task3())
dag = generic_wrapper()
if __name__ == "__main__":
dag.test()
### Operating System
linux
### Versions of Apache Airflow Providers
airflow 2.5.1
### Deployment
Other
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hussein-awala commented on issue #30021: Nested Taskgroup and Dynamic task mapping fails on second group
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on issue #30021:
URL: https://github.com/apache/airflow/issues/30021#issuecomment-1464932530
This issue duplicates #29339.
As I mentioned in my comment there, tasks and task groups expansion in an expanded task group is not supported yet.
If you want to discuss it more, or propose a new design to support nested expansion, you can join this discussion #29618
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hussein-awala closed issue #30021: Nested Taskgroup and Dynamic task mapping fails on second group
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala closed issue #30021: Nested Taskgroup and Dynamic task mapping fails on second group
URL: https://github.com/apache/airflow/issues/30021
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #30021: Nested Taskgroup and Dynamic task mapping issue
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #30021:
URL: https://github.com/apache/airflow/issues/30021#issuecomment-1463690962
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org