You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@airflow.apache.org by Reed Villanueva <rv...@ucera.org> on 2019/11/06 18:50:53 UTC

Import custom python modules into dag file without mixing dag environs and sys.path?

Is there any way to import custom python modules into dag file without
mixing dag environs and sys.path? Can't use something like

environ["PROJECT_HOME"] = "/path/to/some/project/files"# import
certain project files
sys.path.append(environ["PROJECT_HOME"])import mymodule

because it the sys.path is shared among all dags and this causes problems
<https://stackoverflow.com/q/58423137/8236733> (eg. sharing of values
between dag definitions) if want to import modules from different places
that have the same name for different dag definitions (and if there are
many dags, this is hard to keep track of).

The docs for using packaged dags
<https://airflow.apache.org/concepts.html?highlight=zip#packaged-dags> (which
seemed like a solution) do not seem to avoid the problem

the zip file will be inserted at the beginning of module search list
(sys.path) and as such it will be available to any other code that resides
within the same interpreter.

Anyone with more airflow knowledge know how to handle this kind of
situation?

-- 
This electronic message is intended only for the named 
recipient, and may 
contain information that is confidential or 
privileged. If you are not the 
intended recipient, you are 
hereby notified that any disclosure, copying, 
distribution or 
use of the contents of this message is strictly 
prohibited. If 
you have received this message in error or are not the 
named
recipient, please notify us immediately by contacting the 
sender at 
the electronic mail address noted above, and delete 
and destroy all copies 
of this message. Thank you.