[GitHub] [airflow] MeiK2333 commented on issue #5709: [AIRFLOW-5096] use modification time replace last loaded time, reduce database insert

[GitBox <gi...@apache.org>]

MeiK2333 commented on issue #5709: [AIRFLOW-5096] use modification time replace last loaded time, reduce database insert
URL: https://github.com/apache/airflow/pull/5709#issuecomment-517644344
 
 
   > I agree that this is a problem, but your change would break for dynamic DAGs (which is where a DAG .py file generates tasks based on something other than the content of the file, such as a DB query or a config file.)
   > 
   > I think for your case the fix is to turn off dag pickling, by setting `donot_pickle = True` in the `[core]` section of the config.
   
   Thank you for your reply, this is indeed a proven solution, but then we lose the convenience that can be brought to us.
   When this option is enabled, most of the data inserted into the database is not associated with other data, and the data can no longer be read. If we can't modify Dag, can we modify the way DagPickle is generated to circumvent this problem?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services