You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@airflow.apache.org by Reed Villanueva <rv...@ucera.org> on 2019/11/06 18:50:53 UTC
Import custom python modules into dag file without mixing dag
environs and sys.path?
Is there any way to import custom python modules into dag file without
mixing dag environs and sys.path? Can't use something like
environ["PROJECT_HOME"] = "/path/to/some/project/files"# import
certain project files
sys.path.append(environ["PROJECT_HOME"])import mymodule
because it the sys.path is shared among all dags and this causes problems
<https://stackoverflow.com/q/58423137/8236733> (eg. sharing of values
between dag definitions) if want to import modules from different places
that have the same name for different dag definitions (and if there are
many dags, this is hard to keep track of).
The docs for using packaged dags
<https://airflow.apache.org/concepts.html?highlight=zip#packaged-dags> (which
seemed like a solution) do not seem to avoid the problem
the zip file will be inserted at the beginning of module search list
(sys.path) and as such it will be available to any other code that resides
within the same interpreter.
Anyone with more airflow knowledge know how to handle this kind of
situation?
--
This electronic message is intended only for the named
recipient, and may
contain information that is confidential or
privileged. If you are not the
intended recipient, you are
hereby notified that any disclosure, copying,
distribution or
use of the contents of this message is strictly
prohibited. If
you have received this message in error or are not the
named
recipient, please notify us immediately by contacting the
sender at
the electronic mail address noted above, and delete
and destroy all copies
of this message. Thank you.