You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "multimeric (via GitHub)" <gi...@apache.org> on 2026/02/11 04:27:54 UTC
[I] Integration between `XComArg` and `@task_group` [airflow]
multimeric opened a new issue, #61758:
URL: https://github.com/apache/airflow/issues/61758
### Description
Task groups are allowed to return a `DAGNode`:
https://github.com/apache/airflow/blob/752419dae9d27419019395b1aaf9ea1bf497b5e9/task-sdk/src/airflow/sdk/definitions/decorators/task_group.py#L204-L206
https://github.com/apache/airflow/blob/752419dae9d27419019395b1aaf9ea1bf497b5e9/task-sdk/src/airflow/sdk/definitions/decorators/task_group.py#L55
A typical `DAGNode` is `Operator`. However in taskflow you typically work with `XComArg`, not `Operator`.
```python
from typing import reveal_type
from airflow.sdk import dag, task, task_group
@task_group
def some_group() -> Operator:
@task
def generate_data() -> list[int]:
return [1, 2, 3, 4, 5]
result = generate_data()
reveal_type(result)
```
```
XComArg
```
To pull a `DAGNode` out from taskflow, you could do something like this, but it's digging into the internals:
```python
if isinstance(result, PlainXComArg):
return result.operator
else:
raise ValueError("Expected a PlainXComArg, got something else.")
```
The best solution would be to allow the `@task_group` decorated function to return an `XComArg`, and then the `DAGNode` could be inferred from there. Failing that, just a consistent API for retrieving the operator from an `XComArg` would be ideal.
***
The other issue is at the other end of the task group: when you use it.
```python
@dag(
start_date=datetime(2024, 1, 1),
)
def workflow():
group_result = some_group()
reveal_type(group_result)
```
```
DAGNode
```
This is the opposite problem. Here we have a `DAGNode`, but taskflow wants to deal with `XComArg`. The hacky workaround is to cast it to `Operator` and then use `.output`, but this is the same problem.
So I think the `@task_group` decorated function would ideally return `XComArg` as well.
### Use case/motivation
This would effectively allow you to use the return value of a `@task_group`. Currently task groups work as a kind of subDAG that can't return values.
### Related issues
_No response_
### Are you willing to submit a PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Integration between `XComArg` and `@task_group` [airflow]
Posted by "multimeric (via GitHub)" <gi...@apache.org>.
multimeric commented on issue #61758:
URL: https://github.com/apache/airflow/issues/61758#issuecomment-5018400451
> You can already return from a task group and use the result in Task Flow afaik?
From memory, doing this will not pass the type checker since `XComArg` (or `dict[str, XComArg]` which is what you're returning) is not a `DAGNode`. If it works as-is, then it means the type annotations can just be updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Integration between `XComArg` and `@task_group` [airflow]
Posted by "multimeric (via GitHub)" <gi...@apache.org>.
multimeric commented on issue #61758:
URL: https://github.com/apache/airflow/issues/61758#issuecomment-3906087606
That's a shame, I would be happy to implement this but not without some input about the design. The biggest challenge is implementing `@task_group` to better integrate with TaskFlow without breaking it for the old DAG API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Integration between `XComArg` and `@task_group` [airflow]
Posted by "Subham-KRLX (via GitHub)" <gi...@apache.org>.
Subham-KRLX commented on issue #61758:
URL: https://github.com/apache/airflow/issues/61758#issuecomment-3882863567
@multimeric i would like to fix this issue can you please assign this to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Integration between `XComArg` and `@task_group` [airflow]
Posted by "multimeric (via GitHub)" <gi...@apache.org>.
multimeric commented on issue #61758:
URL: https://github.com/apache/airflow/issues/61758#issuecomment-3884450578
Sure I can do that, but I don't know if the maintainers agree with this proposal or if it would get merged
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Integration between `XComArg` and `@task_group` [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #61758:
URL: https://github.com/apache/airflow/issues/61758#issuecomment-3911066892
It's really a question of increasing your chances, you have been given an advice on what helps, so it's up to you what you will do with that advice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Integration between `XComArg` and `@task_group` [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #61758:
URL: https://github.com/apache/airflow/issues/61758#issuecomment-3905143810
> Well I don't know if the maintainers agree with this proposal or if it would get merged. I also don't have the power to assign issues.
This you never know for sure. Depends on quality of implementation tests, and actually maintainers looking at it and deciding if this is good idea after seeing the code. You will rarely get an "agreement" on pretty dry description - actual code is always best to discuss things. And yes, it might seem as unneeded effort, but no - it means that you might learn a lot by doing it, even change your approach and find better solutions while doing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Integration between `XComArg` and `@task_group` [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #61758:
URL: https://github.com/apache/airflow/issues/61758#issuecomment-3911061899
> That's a shame, I would be happy to implement this but not without some input about the design. The biggest challenge is implementing `@task_group` to better integrate with TaskFlow without breaking it for the old DAG API.
Maybe someone will provide it regardless, but again - you never now, but again - discussion on the code - even if not perfect, and with todos and with questions done in the PR are far more likely to attract maintainer to discuss it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Integration between `XComArg` and `@task_group` [airflow]
Posted by "Desdroid (via GitHub)" <gi...@apache.org>.
Desdroid commented on issue #61758:
URL: https://github.com/apache/airflow/issues/61758#issuecomment-4797818063
You can already return from a task group and use the result in Task Flow afaik?
At least we use it like this:
```python
from airflow.sdk import dag, task, task_group
@dag()
def test():
@task()
def add(a, b):
return a + b
@task
def subtract(a, b):
return a - b
@task
def log(something):
print(something)
@task_group
def group(a, b):
result_1 = add(a, b)
result_2 = subtract(a, b)
return {"subtraction": result_2, "sum": result_1}
results = group(20, 5)
log(results["subtraction"])
log(results["sum"])
instance = test()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org