Update airflow dag daily/periodically based on dagfile definition code?

[Reed Villanueva <rv...@ucera.org>]

Is there a way to update airflow dag daily/periodically based on dagfile
definition code? Eg. to update date values that may be used in the dag
definition.

For context: I have an airflow dag that gets new table rows each day from a
remote DB and moves them into a local DB. In order to get the latest rows
from the remote, we have a function that gets the latest date from the
local. Currently have a dag defined like...

...def get_latest_date(tablename):
    # get latest import date from local table
    ....
for table in tables: # type list(dict)

    task_1 = BashOperator(
        task_id='task_1_%s' % table["tablename"],
        bash_command='bash %s/task_1.sh %s' % (PROJECT_HOME,
table["latest_date"]),
        execution_timeout=timedelta(minutes=30),
        dag=dag)

    task_2 = BashOperator(
        task_id='task_2_%s' % table["tablename"],
        bash_command='bash %s/task_2.sh' % PROJECT_HOME,
        execution_timeout=timedelta(minutes=30),
        dag=dag)

    task_1 >> task_2

where tables are dicts where one of their fields in constructed earlier in
the code to be a string rep of the latest date for a given table. When
printing the supposed latest date in the task_1.sh script, finding that the
date does not update each day. Need a way for the tables list be be built
anew each day to have the right date values.

-- 
This electronic message is intended only for the named 
recipient, and may 
contain information that is confidential or 
privileged. If you are not the 
intended recipient, you are 
hereby notified that any disclosure, copying, 
distribution or 
use of the contents of this message is strictly 
prohibited. If 
you have received this message in error or are not the 
named
recipient, please notify us immediately by contacting the 
sender at 
the electronic mail address noted above, and delete 
and destroy all copies 
of this message. Thank you.