Use separate environ and sys.path between dags

[Reed Villanueva <rv...@ucera.org>]

Is there a way to have dags use separate environ and sys.path values?

For example, I have files dag1.py and dag2.py that both want to import
modules and run files as tasks from directories /files/for/dag1 and
/files/for/dag2. The files are named the same in both dirs but do very
different things. Currently, if I set environ (eg. PROJECT_HOME) and
sys.path in one dag file, I see that it affects all other dag files in
~/airflow/dag. This causes problems / mixups when trying to import modules
or reference files in dirs where files have same names.

Is there a way to get this working without using packaged dags? Can't use
packaged dags, since have some bash scripts that run as tasks that use
environ variables (like PROJECT_HOME) to reference local dirs for other
things and as I understand it a packaged dag will only help co-locating the
files, but not isolate env vars.

-- 
This electronic message is intended only for the named 
recipient, and may 
contain information that is confidential or 
privileged. If you are not the 
intended recipient, you are 
hereby notified that any disclosure, copying, 
distribution or 
use of the contents of this message is strictly 
prohibited. If 
you have received this message in error or are not the 
named
recipient, please notify us immediately by contacting the 
sender at 
the electronic mail address noted above, and delete 
and destroy all copies 
of this message. Thank you.

Re: Use separate environ and sys.path between dags

[Reed Villanueva <rv...@ucera.org>]

Interesting, thanks for the info.

On Thu, Oct 17, 2019 at 12:11 PM Kamil Breguła <ka...@polidea.com>
wrote:

> Airflow does not have a built-in mechanism, but if you have the
> opportunity to create a contribution then you should look in the
> direction of BaseTaskRunner.
>
> On Thu, Oct 17, 2019 at 9:57 PM Reed Villanueva <rv...@ucera.org>
> wrote:
> >
> > Is there a way to have dags use separate environ and sys.path values?
> >
> > For example, I have files dag1.py and dag2.py that both want to import
> modules and run files as tasks from directories /files/for/dag1 and
> /files/for/dag2. The files are named the same in both dirs but do very
> different things. Currently, if I set environ (eg. PROJECT_HOME) and
> sys.path in one dag file, I see that it affects all other dag files in
> ~/airflow/dag. This causes problems / mixups when trying to import modules
> or reference files in dirs where files have same names.
> >
> > Is there a way to get this working without using packaged dags? Can't
> use packaged dags, since have some bash scripts that run as tasks that use
> environ variables (like PROJECT_HOME) to reference local dirs for other
> things and as I understand it a packaged dag will only help co-locating the
> files, but not isolate env vars.
> >
> > This electronic message is intended only for the named
> > recipient, and may contain information that is confidential or
> > privileged. If you are not the intended recipient, you are
> > hereby notified that any disclosure, copying, distribution or
> > use of the contents of this message is strictly prohibited. If
> > you have received this message in error or are not the named
> > recipient, please notify us immediately by contacting the
> > sender at the electronic mail address noted above, and delete
> > and destroy all copies of this message. Thank you.
>

-- 
This electronic message is intended only for the named 
recipient, and may 
contain information that is confidential or 
privileged. If you are not the 
intended recipient, you are 
hereby notified that any disclosure, copying, 
distribution or 
use of the contents of this message is strictly 
prohibited. If 
you have received this message in error or are not the 
named
recipient, please notify us immediately by contacting the 
sender at 
the electronic mail address noted above, and delete 
and destroy all copies 
of this message. Thank you.

Re: Use separate environ and sys.path between dags

[Kamil Breguła <ka...@polidea.com>]

Airflow does not have a built-in mechanism, but if you have the
opportunity to create a contribution then you should look in the
direction of BaseTaskRunner.

On Thu, Oct 17, 2019 at 9:57 PM Reed Villanueva <rv...@ucera.org> wrote:
>
> Is there a way to have dags use separate environ and sys.path values?
>
> For example, I have files dag1.py and dag2.py that both want to import modules and run files as tasks from directories /files/for/dag1 and /files/for/dag2. The files are named the same in both dirs but do very different things. Currently, if I set environ (eg. PROJECT_HOME) and sys.path in one dag file, I see that it affects all other dag files in ~/airflow/dag. This causes problems / mixups when trying to import modules or reference files in dirs where files have same names.
>
> Is there a way to get this working without using packaged dags? Can't use packaged dags, since have some bash scripts that run as tasks that use environ variables (like PROJECT_HOME) to reference local dirs for other things and as I understand it a packaged dag will only help co-locating the files, but not isolate env vars.
>
> This electronic message is intended only for the named
> recipient, and may contain information that is confidential or
> privileged. If you are not the intended recipient, you are
> hereby notified that any disclosure, copying, distribution or
> use of the contents of this message is strictly prohibited. If
> you have received this message in error or are not the named
> recipient, please notify us immediately by contacting the
> sender at the electronic mail address noted above, and delete
> and destroy all copies of this message. Thank you.