[GitHub] [airflow] SamWheating commented on pull request #17963: Improve the description of how to handle dynamic task generation

[GitBox <gi...@apache.org>]

SamWheating commented on pull request #17963:
URL: https://github.com/apache/airflow/pull/17963#issuecomment-910417229


   I've encountered some race-conditions with this kind of dynamic task generation before, typically when trying to update the metadata file and trigger the DAG immediately after. 
   
   Two cases where this shows up:
   
    - DAG_1 writes a metadata file and triggers DAG 2 (which uses the metadata file to generate tasks)
    - Uploading a metadata file and immediately triggering a DAG via REST API
    
   In these cases, the DAG often isn't re-processed between the time that the metadata file is updated and the time that the DAG is triggered, so the tasks which are created are based on the previous version of the metadata file.
   
   I think that this can be a bit of a trap and can definitely be misleading for users who aren't familiar with Airflow's internals. Do you think we can include some warnings about this behaviour in the documentation?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org