[GitHub] [airflow] dzhigimont commented on issue #29531: Dynamic task mapping does not always create mapped tasks

["dzhigimont (via GitHub)" <gi...@apache.org>]

dzhigimont commented on issue #29531:
URL: https://github.com/apache/airflow/issues/29531#issuecomment-1523977469

   Hi @potiuk, I've prepared the MR https://github.com/apache/airflow/pull/30892 with a solution to such a problem.
   The previous fix doesn't work for two reasons:
   1.  dependecies are checked before expanding of mapped tasks
   2. amount of mapped tasks for each dag run can be different so when current dag run have greater amount of tasks then previous dag run the depends_on_past dependecy will never be sutisfied.
   
   I propose a solution in which I check all previously mapped tasks for one task_id as one whole dependency for the current mapped task since we can have different amounts of mapped tasks, and if the current dag run has more mapped tasks and there is no such index in the previous dag run, it is difficult to understand that the previous dag run really doesn't have such an index or tasks haven't been expanded yet. Also, I think if tasks have the same map_index, it aren't guaranteed that they do the same work and it's better to wait until all mapped tasks in the past finish.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org