You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/05/12 13:15:14 UTC
[GitHub] [airflow] uranusjr commented on issue #23621: using `expand` inside @task_group will cause a `airflow.exceptions.DuplicateTaskIdFound`
uranusjr commented on issue #23621:
URL: https://github.com/apache/airflow/issues/23621#issuecomment-1124980797
> It is notable that `MappedOperator` actually doesn't seem to care about logically separating the task mappings using the map_index, so as far as airflow knows they are perfect copies of the same task instance, hence, at the minimum attempt of nesting a mapped task somewhere, it goes haywire.
An instance of a _task_ and a _task instance_ are two different concepts in Airflow (it’s super confusing I know). A task is a thing in a DAG (think about the boxes you see in the Graph View), while a task instance is something that gets executed inside a worker. A mapped task spawns multiple task instances to run in the worker, but those task instances belong to the same logical task, which doesn’t have a notion of map index, but only know whether itself is being mapped (MappedOperator) or not.
In your DAG attached above, each loop iteration creates a new _task_ from `task_2` that is a mapped _task_ and can be expanded into multiple _task instances_. Task instances expanded from the same task do share an identical task, but that’s not the issue here; the issue is the DAG needs to differentiate different _tasks_ created from `task_2`, via some mechanism that is not currently implemented correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org