You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/04/23 10:06:36 UTC

[GitHub] [airflow] KayleMaster opened a new issue #8528: DAG that uses data from config files will never update unless touched

KayleMaster opened a new issue #8528:
URL: https://github.com/apache/airflow/issues/8528


   **Description**
   
   Provide some sort of mechanism for airflow to periodically update DAGs which use data from config files. This could be some sort of flag in the DAG constructor so we don't have to monitor every DAG.
   
   **Use case / motivation**  
   
   We have a development and a production environment. The only difference is the config files - these determine where the data is exported/imported from. In production, when we updated the config file, the DAG wouldn't update for an hour. We couldn't wait more and touched the dag file in order to update with the new config values.
   
   **Related Issues**
   
   None that I could find.  
   
     
   Please tell me if there's any workarounds. The config file is outside of the airflow directory. If I symlink it in the same folder the dag resides in, would that update it?
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ashb closed issue #8528: DAG that uses data from config files will never update unless touched

Posted by GitBox <gi...@apache.org>.
ashb closed issue #8528:
URL: https://github.com/apache/airflow/issues/8528


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #8528: DAG that uses data from config files will never update unless touched

Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #8528:
URL: https://github.com/apache/airflow/issues/8528#issuecomment-833526548


   Can you please elaborate on the use case?
   I'm a bit confused.
   If the configuration file construct the DAG structure then it needs to be called in a top level code thus the configuration file will be parsed every time airflow parse the dag.
   If the configuration file construct a task then it will be used on run time.
   Either way you will have the updated file.
   
   But if I misunderstood your scenario still it sounds more like a use case for services like aws lambda - where you execute a function (refresh dag code) when someone updates the configuration file.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ashb commented on issue #8528: DAG that uses data from config files will never update unless touched

Posted by GitBox <gi...@apache.org>.
ashb commented on issue #8528:
URL: https://github.com/apache/airflow/issues/8528#issuecomment-833565025


   The default configuration for Airflow will re-parse the dag file every 30s (https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#min-file-process-interval) -- or try to. If you have lots of dags it might take longer.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal edited a comment on issue #8528: DAG that uses data from config files will never update unless touched

Posted by GitBox <gi...@apache.org>.
eladkal edited a comment on issue #8528:
URL: https://github.com/apache/airflow/issues/8528#issuecomment-833526548


   Can you please elaborate on the use case?
   I'm a bit confused.
   If the configuration file construct the DAG structure then it needs to be called in a top level code thus the configuration file will be parsed every time airflow parse the dag.
   If the configuration file construct a task then it will be used on run time.
   Either way you will have the updated file.
   
   But if I misunderstood your scenario still it sounds more like a use case for services like aws lambda - where you execute a function (refresh dag file) when someone updates the configuration file.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org