You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/11/30 19:44:58 UTC
[GitHub] [airflow] jceresini opened a new issue #19902: DuplicateTaskIdFound when reusing tasks with task_group decorator
jceresini opened a new issue #19902:
URL: https://github.com/apache/airflow/issues/19902
### Apache Airflow version
2.2.2 (latest released)
### Operating System
MacOS 11.6.1
### Versions of Apache Airflow Providers
$ pip freeze | grep airflow
apache-airflow==2.2.2
apache-airflow-providers-celery==2.1.0
apache-airflow-providers-ftp==2.0.1
apache-airflow-providers-http==2.0.1
apache-airflow-providers-imap==2.0.1
apache-airflow-providers-sqlite==2.0.1
### Deployment
Other
### Deployment details
`airflow standalone`
### What happened
Exception raised :
```
raise DuplicateTaskIdFound(f"Task id '{key}' has already been added to the DAG")
airflow.exceptions.DuplicateTaskIdFound: Task id 'do_all_things.do_thing__1' has already been added to the DAG
```
### What you expected to happen
_No response_
### How to reproduce
* Add the following python file to the `dags` folder:
```python
from datetime import datetime
from airflow.decorators import dag, task, task_group
@dag(start_date=datetime(2023, 1, 1), schedule_interval="@once")
def test_dag_1():
@task
def start():
pass
@task
def do_thing(x):
print(x)
@task_group
def do_all_things():
for i in range(5):
do_thing(i)
@task
def end():
pass
start() >> do_all_things() >> end()
test_dag_1()
```
* Start airflow by running `airflow standalone`
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #19902: DuplicateTaskIdFound when reusing tasks with task_group decorator
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #19902:
URL: https://github.com/apache/airflow/issues/19902#issuecomment-982961656
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #19902: DuplicateTaskIdFound when reusing tasks with task_group decorator
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #19902:
URL: https://github.com/apache/airflow/issues/19902#issuecomment-1012992340
Oh the cause is simpler than I though; fixing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr closed issue #19902: DuplicateTaskIdFound when reusing tasks with task_group decorator
Posted by GitBox <gi...@apache.org>.
uranusjr closed issue #19902:
URL: https://github.com/apache/airflow/issues/19902
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jceresini commented on issue #19902: DuplicateTaskIdFound when reusing tasks with task_group decorator
Posted by GitBox <gi...@apache.org>.
jceresini commented on issue #19902:
URL: https://github.com/apache/airflow/issues/19902#issuecomment-982971847
I just noticed that the above works if I ditch the for loop in the _task_group_, and run each _task_ separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #19902: DuplicateTaskIdFound when reusing tasks with task_group decorator
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #19902:
URL: https://github.com/apache/airflow/issues/19902#issuecomment-1012840961
(Self-assigning so maybe I can come back to it at some point. But feel free to raise your hand if you want to tak a stab instead!)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #19902: DuplicateTaskIdFound when reusing tasks with task_group decorator
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #19902:
URL: https://github.com/apache/airflow/issues/19902#issuecomment-1012840620
Yeah, this is a known problem (well, I know it, at least) with `@task`, you need to instead do something like
```python
@task_group
def do_all_things():
for i in range(5):
@task(task_id=f"do_thing_{i}")
def do_thing(i):
print(i)
do_thing(i)
```
instead. But it’s unfortunately not very easy to fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org