[GitHub] [airflow] potiuk commented on issue #29555: Dag Fails while creating a Dynamic Tasks using airflow variables.

["potiuk (via GitHub)" <gi...@apache.org>]

potiuk commented on issue #29555:
URL: https://github.com/apache/airflow/issues/29555#issuecomment-1439933296

   I believe (If I understand correctly) you cannot use variables at the top-level to dynamically add tasks like that. The structure of the  DAG has to be fixed when it is parsed, It cannot dynamically create different number of tasks like that between runs. You can "slowly"  change the number of tasks when you construct your DAG based on some external data, but it cannot widely differ between the moment when you parsed it and the moment you executed it.
   
   If you want to have dynamic number of tasks changing between runs you have to use Dynamic Task Mapping:  https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/dynamic-task-mapping.html#dynamic-task-mapping - where you will have one task that will produce the "batch_list" (It could be done from variables) and then you should use expand() method to expand the task lkist dynamically at runtime (what you are doing here you try to dynamically change the structure of DAG at the moment it is parsed, not at the moment it is executed).
   
   Closing provisionally, we can reopen if I understood it wrongly


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org