[GitHub] [airflow] kaxil commented on a diff in pull request #24518: Re-serialize all DAGs on 'airflow db upgrade'

[GitBox <gi...@apache.org>]

kaxil commented on code in PR #24518:
URL: https://github.com/apache/airflow/pull/24518#discussion_r903972985


##########
airflow/utils/db.py:
##########
@@ -788,6 +788,14 @@ def check_and_run_migrations():
         sys.exit(1)
 
 
+@provide_session
+def reserialize_dags(*, session: Session = NEW_SESSION) -> None:
+    session.query(SerializedDagModel).delete(synchronize_session=False)
+    dagbag = DagBag()
+    dagbag.collect_dags(only_if_updated=False, safe_mode=False)
+    dagbag.sync_to_db(session=session)
+

Review Comment:
   At the very least I would really like to have a flag to not do that.
   
   I would like to control when DAGs get reserialized. For customers with bigger DAGs (more than >100 tasks) and larger number of DAGs (>100-500)< this is good to add a quite bit of time. Some of our migrations already take a good chunk of time, I would really like to avoid that here. " Time to Parse the DAGs + Time to serialize it + Send the entire blob over the network).
   
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org