[GitHub] [airflow] potiuk commented on a diff in pull request #24518: Re-serialize all DAGs on 'airflow db upgrade'

[GitBox <gi...@apache.org>]

potiuk commented on code in PR #24518:
URL: https://github.com/apache/airflow/pull/24518#discussion_r909279179


##########
airflow/utils/db.py:
##########
@@ -788,6 +788,14 @@ def check_and_run_migrations():
         sys.exit(1)
 
 
+@provide_session
+def reserialize_dags(*, session: Session = NEW_SESSION) -> None:
+    session.query(SerializedDagModel).delete(synchronize_session=False)
+    dagbag = DagBag()
+    dagbag.collect_dags(only_if_updated=False, safe_mode=False)
+    dagbag.sync_to_db(session=session)
+

Review Comment:
   I am not sure, but i think we should know if we are talking about 5% of time increase (for example when we add simple new field to a table with task instance) or 50% or 200%. Or put it another way - are we talking about 5 minutes or 1 hour of overhead. 
   
   I think we should get some data to base our decisions on. I believe (this is rather my intuition than hard data) that with  even a big installation re-serialization of DAGs will not take a long time - minutes at most. 
   
   Back-of-the envelope calcilations. Cardinality of SerializedDAG table is (until we add versioning at least) the smallest when it comes to "dag-related" tables (and this is the most important factor IMHO). Number of rows in the table is equal to number of DAGs. So it's not like we multiple 500 DAGs by 100 tasks in the case above. For comparision - Task Instance is by far the worst, because we have DAG num * per-dag task num * number of runs. This can grow fast, quickly, but not SerializedDAG table. Yeah. You have to parse all those files. But even eve if you take 0.5s to parse each DAG, with 500 DAGs you get 6 minutes. And 0.5 per DAG indicates that the user has far bigger problems already. Sending 500 blobs (this is what reserializing does) and deleting/updating 500 rows in a table is unlikely to be slow. Sending biggish blobs one per transactions is about as efficient as it gets when it comes to overhead of communication with the DB - roughly equivalent to sending the raw data (o
 verhead for transaction and row transaction is not neglectible but it is order of magnitude less than sending the data). Assuming we have 1MB Blob to send per DAG (which is outrageous) we are talking about sending roughly 500 MB to the DB. Assuming remote databse and 100 Mb connection (extremely low for a prod db): 2 minutes.
   
   So back-of -the envelope calculation tells me that even with such huge installation, we are talking about at most few minutes overhead.
   
   My intuition tells me that if this is 5% or 5 minutes, this is entirely justifiable to add delay. Conversely - It will take  hours of downtime and investigation and loosing time of multiple people to deal with serialization issue when it happens, so avoiding "sure" one-time 5-minutes delay once-a-few-months in favour of possible hours of investigations is simply premature optimization. There are always cost coming with optimizations - complexity, quite likely risk of failure in this case. 
   
   On the other hand if for some customers it will add hours of delays, then yeah. we might think about optimizing it and avoiding the reserialization.
   
   Do we actually have any number on how much does it take for a customer that hass 500 dags and 100 tasks? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org