You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "LiamOrmiston (via GitHub)" <gi...@apache.org> on 2023/08/02 21:43:04 UTC
[GitHub] [airflow] LiamOrmiston opened a new issue, #33053: ExternalTaskSensor waits forever if TaskGroup contains skipped tasks
LiamOrmiston opened a new issue, #33053:
URL: https://github.com/apache/airflow/issues/33053
### Apache Airflow version
Other Airflow 2 version (please specify below)
### What happened
If you have an `ExternalTaskSensor` that uses `external_task_group_id` to wait on a `TaskGroup`, and if that `TaskGroup` contains any skipped tasks, the sensor will be stuck waiting forever despite the UI saying the state of the `TaskGroup` is successful.
### What you think should happen instead
`ExternalTaskSensor` should match the UI's interpretation of the `TaskGroup` state.
### How to reproduce
```
#!/usr/bin/env python3
import datetime
import logging
from airflow.decorators import dag, task
from airflow.operators.empty import EmptyOperator
from airflow.sensors.external_task import ExternalTaskSensor
from airflow.utils.task_group import TaskGroup
from airflow.exceptions import AirflowFailException, AirflowSkipException
logger = logging.getLogger(__name__)
@dag(
schedule_interval='@daily',
start_date=datetime.datetime(2023, 8, 1),
)
def task_groups():
with TaskGroup(group_id='skip_group'):
@task
def skip_task():
raise AirflowSkipException
EmptyOperator(task_id='operator1') >> skip_task()
with TaskGroup(group_id='pass_group'):
@task
def pass_task():
pass
EmptyOperator(task_id='operator3') >> pass_task()
ExternalTaskSensor(
task_id='wait_for_task_group_with_skipped_task',
external_dag_id='task_groups',
external_task_group_id='skip_group',
check_existence=True,
)
ExternalTaskSensor(
task_id='wait_for_task_group_with_passed_task',
external_dag_id='task_groups',
external_task_group_id='pass_group',
check_existence=True,
)
dag = task_groups()
if __name__ == '__main__':
dag.cli()
```
### Operating System
CentOS Stream 8
### Versions of Apache Airflow Providers
_No response_
### Deployment
Other
### Deployment details
Standalone
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #33053: ExternalTaskSensor waits forever if TaskGroup contains skipped tasks
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #33053:
URL: https://github.com/apache/airflow/issues/33053#issuecomment-1685083915
Yep. Very much on it with what @hussein-awala wrote.
The TaskGroup is not **YET** a fullly featured "execution" construct in Airflow. From the very beginning it has been a UI-only feature, but over the last few minor releases it has gotten more of the "execution construct" features (cleaning task group, setup/teardown for example) and I think we need to add a few more things to make it "first-class" citizen that will be able to replace the deprecated SubDag (and much more).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Gollum999 commented on issue #33053: ExternalTaskSensor waits forever if TaskGroup contains skipped tasks
Posted by "Gollum999 (via GitHub)" <gi...@apache.org>.
Gollum999 commented on issue #33053:
URL: https://github.com/apache/airflow/issues/33053#issuecomment-1684348348
IMO what matters most is that the behavior is consistent. If a TaskGroup behaves like a skipped task when it comes to things like `allowed_states` and `trigger_rule`s, it should show as skipped in the UI. And vice-versa; if a TaskGroup shows as success in the UI, it should also behave like a successful task everywhere else.
From an implementation perspective, it feels to me like both the scheduler and the webserver should be using the same code to determine the TaskGroup's "pseudo-state".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] nathadfield commented on issue #33053: ExternalTaskSensor waits forever if TaskGroup contains skipped tasks
Posted by "nathadfield (via GitHub)" <gi...@apache.org>.
nathadfield commented on issue #33053:
URL: https://github.com/apache/airflow/issues/33053#issuecomment-1680418631
@LiamOrmiston I think what this highlights is the fact that a TaskGroup is **only** really a UI construct and, when checking upstream dependencies, this is based upon checking the status of all the individual tasks within the group rather than the group. Task Groups don't actually have any state of their own.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] LiamOrmiston commented on issue #33053: ExternalTaskSensor waits forever if TaskGroup contains skipped tasks
Posted by "LiamOrmiston (via GitHub)" <gi...@apache.org>.
LiamOrmiston commented on issue #33053:
URL: https://github.com/apache/airflow/issues/33053#issuecomment-1664671806
Thank you for your reply, @dzhigimont. To clarify, I understand there is a way to get the behavior I expect. My main intention for this issue _is_ to point out the differences in the default inpretation of states between the UI and the `ExternalTaskSensor`.
Currently, the UI interpretation for `TaskGroup` state matches the UI interpretation for a [DagRun](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dag-run.html#dag-run-status); that is, a `DagRun` can be successful with `skipped` and `success` task states. Because of this, it is natural to assume that `ExternalTaskSensor` would behave the same if it was a sensor for a `TaskGroup` that contains a skipped task or a sensor for a `DagRun` that contains a skipped task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dzhigimont commented on issue #33053: ExternalTaskSensor waits forever if TaskGroup contains skipped tasks
Posted by "dzhigimont (via GitHub)" <gi...@apache.org>.
dzhigimont commented on issue #33053:
URL: https://github.com/apache/airflow/issues/33053#issuecomment-1664458811
Hi, @LiamOrmiston, there is no need to open the issues because `ExternalTaskSensor` already has a parameter to achieve what you want
`:param allowed_states: Iterable of allowed states, default is ['success']`
So by default, `ExternalTaskSensor` waits until all tasks in the group will be in `success` state but you can add `skipped` state as well
```
ExternalTaskSensor(
task_id='wait_for_task_group_with_skipped_task',
external_dag_id='task_groups_lala',
external_task_group_id='skip_group',
check_existence=True,
allowed_states=[State.SUCCESS, State.SKIPPED],
)
```
I think how UI shows the status for groups it's a different question
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ExternalTaskSensor waits forever if TaskGroup contains skipped tasks [airflow]
Posted by "guhuajun (via GitHub)" <gi...@apache.org>.
guhuajun commented on issue #33053:
URL: https://github.com/apache/airflow/issues/33053#issuecomment-2122113434
Greetings,
I am leveraging my Celery project experience to understand Airflow project.Tried ExternalTaskSensor with two DAGs for several hours, it's failed. Then I switched to `TriggerDagRunOperator`, then it works.
> Yep. Very much on it with what @hussein-awala wrote.
>
> The TaskGroup is not **YET** a fullly featured "execution" construct in Airflow. From the very beginning it has been a UI-only feature, but over the last few minor releases it has gotten more of the "execution construct" features (cleaning task group, setup/teardown for example) and I think we need to add a few more things to make it "first-class" citizen that will be able to replace the deprecated SubDag (and much more).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] nathadfield closed issue #33053: ExternalTaskSensor waits forever if TaskGroup contains skipped tasks
Posted by "nathadfield (via GitHub)" <gi...@apache.org>.
nathadfield closed issue #33053: ExternalTaskSensor waits forever if TaskGroup contains skipped tasks
URL: https://github.com/apache/airflow/issues/33053
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hussein-awala commented on issue #33053: ExternalTaskSensor waits forever if TaskGroup contains skipped tasks
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on issue #33053:
URL: https://github.com/apache/airflow/issues/33053#issuecomment-1685081328
@LiamOrmiston we're discussing in #21867 adding a retry option at the task group level, and this is the first step to no longer consider the task group as just a component of UI, where in the future we will add some configurations and options to the task group class.
IMO it will be possible to add a state for the task group, which you are suggesting.
But currently this is not a bug because it's an expected behavior, where as its name, a task group is a group of tasks have the same task group ID, and the use of a task group in your `ExternalTaskSensor` is equivalent to providing the list of their ids via `external_task_ids`.
I will accept the issue as a request for a new feature, which is adding a state for the task group.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #33053: ExternalTaskSensor waits forever if TaskGroup contains skipped tasks
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #33053:
URL: https://github.com/apache/airflow/issues/33053#issuecomment-1663003960
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org