[GitHub] [airflow] nik-davis commented on pull request #13893: Fix race condition when using Dynamic DAGs

[GitBox <gi...@apache.org>]

nik-davis commented on pull request #13893:
URL: https://github.com/apache/airflow/pull/13893#issuecomment-767403524


   Thanks @kaxil for the fix, we'll try that out today and let you know if there's any issues.
   
   I haven't been able to reproduce the error locally yet - I imagine with a local DB connection the DAG serialization is happening too fast to trigger the race condition. I'm wondering if adding a large file import to one of the operators could artificially slow it down a bit, but I don't know enough about how the serialization works to know for sure.
   
   I've only managed to get the error locally by removing a row from the serialized_dag table. It looks like the try/excepts you've added would handle this though as well.
   
   Here's the script I was toying with locally anyway to try and trigger the error, in case that's of use (increase the value in the range function to generate more DAGs):
   
   ```
   from airflow.decorators import dag, task
   from airflow.utils.dates import days_ago
   
   
   def create_dag(dag_id):
       @dag(
           default_args={"owner": "airflow", "start_date": days_ago(1)},
           schedule_interval="@once",
           dag_id=dag_id,
           catchup=False,
       )
       def dynamic_dag():
           @task()
           def dynamic_task_1(**kwargs):
               return 1
   
           task_1 = dynamic_task_1()
           task_2 = dynamic_task_1(task_1)
   
       return dynamic_dag()
   
   
   for i in range(2):
       dag_id = f"dynamic_dag_{i}"
       globals()[dag_id] = create_dag(dag_id)
   
   ```
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org