You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "yarong-lifemap (via GitHub)" <gi...@apache.org> on 2023/02/06 09:03:42 UTC
[GitHub] [airflow] yarong-lifemap opened a new issue, #29380: Task Group options in Graph view
yarong-lifemap opened a new issue, #29380:
URL: https://github.com/apache/airflow/issues/29380
### Description
Due to the complexity of our DAG, we only use the Graph view to manage it. It would be great if we could get the task group options (specifically clearing a task group - (see #26658, #28003).
### Use case/motivation
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] set92 commented on issue #29380: Task Group options in Graph view
Posted by "set92 (via GitHub)" <gi...@apache.org>.
set92 commented on issue #29380:
URL: https://github.com/apache/airflow/issues/29380#issuecomment-1461279676
1. **TLDR: After thinking about different cases, I think is not really needed.** Old text: Yep, that's right. For example, if this image if something fails in `check_integrity` I would only want to clear from that task forward. Because `analyze` is not idempotent, for example because it inserted some rows in a bigquery table, and if I run it again it would duplicate the data. So, I would like to click `check_integrity` and clear downstream within task group.
- And why not clear downstream globally? Because if `report` is another task group that is already in success (because of another task group not in the image), you wouldn't want to clear it. --- Now that I was writing and thinking about the possible options, probably I could cover my use case with a `clear downstream of failed tasks`. I was trying to think in edge cases, but I can't think of one that you couldn't solve with the current options in Airflow, so I suppose is not really needed. I'm in the v2.4.2, and since we were waiting on the new interface to not update every new version, I haven't been able to try it the current `clear task group` yet.
**Edit**: Well, now that I think again, maybe one edge case would be if the DAG already finished, but you want to clear some part of a task group. In this case, you wouldn't be able to clear the whole task group because you wish to keep some parts of the run, and you wouldn't be able to `clear downstream` because the rest of the task runs are valid.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bbovenzi commented on issue #29380: Task Group options in Graph view
Posted by "bbovenzi (via GitHub)" <gi...@apache.org>.
bbovenzi commented on issue #29380:
URL: https://github.com/apache/airflow/issues/29380#issuecomment-1460988523
1. If clearing a task in a task group, have an option to limit upstream and downstream to only within the task group?
2. I'm working on that in #29851
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yarong-lifemap commented on issue #29380: Task Group options in Graph view
Posted by "yarong-lifemap (via GitHub)" <gi...@apache.org>.
yarong-lifemap commented on issue #29380:
URL: https://github.com/apache/airflow/issues/29380#issuecomment-1420015103
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] set92 commented on issue #29380: Task Group options in Graph view
Posted by "set92 (via GitHub)" <gi...@apache.org>.
set92 commented on issue #29380:
URL: https://github.com/apache/airflow/issues/29380#issuecomment-1460962674
Wasn't sure where to comment it, but this morning was thinking that it would be nice to have the `clear upstream/downstream inside the taskgroups`.
In my use case, typically some task inside a taskgroup fails, so I want to avoid clearing the whole taskgroup because not all tasks are idempotent, or because there were only a couple of tasks left to finish the current taskgroup. So, it would be fantastic if I could say "I want to clear downstream from a certain task until the end of the current taskgroup". But not sure how common is this use case, so maybe it doesn't make sense to implement it.
Also, another cool button I thought was having a `clear current task in the log window`. Because usually when a task fails, I go to check the log of some task. And sometimes the solution is to relaunch or fix something externally, so I wouldn't need to go back to clear the task, is something I could do from inside the log view. It would be like a fast button inside the log view.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bbovenzi commented on issue #29380: Task Group options in Graph view
Posted by "bbovenzi (via GitHub)" <gi...@apache.org>.
bbovenzi commented on issue #29380:
URL: https://github.com/apache/airflow/issues/29380#issuecomment-1419276724
I'm already working on this. Should be in 2.6.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bbovenzi closed issue #29380: Task Group options in Graph view
Posted by "bbovenzi (via GitHub)" <gi...@apache.org>.
bbovenzi closed issue #29380: Task Group options in Graph view
URL: https://github.com/apache/airflow/issues/29380
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org