[GitHub] [airflow] potiuk edited a comment on issue #20196: dags folder mount readonly with git-sync true option

[GitBox <gi...@apache.org>]

potiuk edited a comment on issue #20196:
URL: https://github.com/apache/airflow/issues/20196#issuecomment-992589498


   > The thing is that my dags are very dynamic and can change very often. I'm not worried about their safety, since the dag master takes metadata from the database. Therefore, I see no reason to store them in git. It would be great to be able to the user to change the parameters of these settings depending on his needs.
   
   Then probably you are doing it wrong (and it will not work as you expect them). From the main Airflow page:
   https://airflow.apache.org/docs/apache-airflow/stable/index.html
   
   
   > Workflows are expected to be mostly static or slowly changing. You can think of the structure of the tasks in your workflow as slightly more dynamic than a database structure would be. Airflow workflows are expected to look similar from a run to the next, this allows for clarity around unit of work and continuity.
   
   You should rethink if Airflow is a good choice for you if you want the DAGs to change frequently. The fact that you generate file in DAG folder does not mean that it will be reflected immediately or even very quickly in the DAG structure (it needs to be parsed by DagParser, stored in serialized dag table and scheduled by scheduler). Also it means that  your history of DAG is correct. And most of all - you will never be sure which version of DAG is run at any given time.
   
   DAG Versioning which is on the horizon of  Airflow (but no-one actively works on it) is supposed to handle this case but it's monhs if not years away: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-36+DAG+Versioning
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org