serialization of tasks within DAG run

[Luke Rohde <lr...@quartethealth.com>]

Hello Airflowians!

At the recent airflow meetup in NYC at Blue Apron, I talked with a few
folks (Andrew, Jeremiah) about a question that had been on my mind: if one
deploys code in the middle of a DAG run, can't it be that the completed
parts of the DAG will have run a different version of the code from the
incomplete parts? The DAG itself is serialized and stored in the database
when using remote executors, but this is not a complete solution, since the
code backing the tasks could be different. It seems to me that workers
shouldn't even necessarily need direct access to code as it is deployed;
just as they are deserializing the DAG from the DB, they should be able to
deserialize the task (or obtain it via some other consistent source) and
run it. Jeremiah mentioned that he recalled someone looking into this in
the past and hitting a wall with jinja templates' pickle-ability.

Curious what other people think about this issue and whether the
maintainers would be interested in a patch attempting to fix this - I'd be
happy to take a stab if so.

-Luke