You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@airflow.apache.org by Reed Villanueva <rv...@ucera.org> on 2019/10/16 00:03:02 UTC
How to manage python packages between airflow dags?
If I have multiple airflow dags with some overlapping python package
dependencies, how can I keep each of these project deps. decoupled? Eg. if
I had project A and B on same server I would run each of them with
something like...
source /path/to/virtualenv_a/activate
python script_a.py
deactivate
source /path/to/virtualenv_b/activate
python script_b.py
deactivate
Basically, would like to run dags with the same situation (eg. each dag
uses python scripts that have may have overlapping package deps. that I
would like to develop separately (ie. not have to update all code using a
package when want to update the package just for one project)). Note, I've
been using the BashOperator to run python tasks like...
do_stuff = BashOperator(
task_id='my_task',
bash_command='python /path/to/script.py'),
execution_timeout=timedelta(minutes=30),
dag=dag)
Is there a way to get this working? Is there some other best-practice way
that airflow intendeds for people to address (or avoid) these kinds of
problems?
--
This electronic message is intended only for the named
recipient, and may
contain information that is confidential or
privileged. If you are not the
intended recipient, you are
hereby notified that any disclosure, copying,
distribution or
use of the contents of this message is strictly
prohibited. If
you have received this message in error or are not the
named
recipient, please notify us immediately by contacting the
sender at
the electronic mail address noted above, and delete
and destroy all copies
of this message. Thank you.
Re: How to manage python packages between airflow dags?
Posted by Reed Villanueva <rv...@ucera.org>.
Good idea, thanks.
Works well for the modular way I am developing python scripts and modules.
On Tue, Oct 15, 2019 at 11:19 PM Kamil Breguła <ka...@polidea.com>
wrote:
> Don't use a wrapper, because only what it does changes the value of
> the PATH variable.
> 11:16 $ type python
> python is /Library/Frameworks/Python.framework/Versions/2.7/bin/python
> 11:16 $ workon airflow
> 11:16 $ type python
> python is /Users/kamilbregula/.virtualenvs/airflow/bin/python
>
> so if you want to use a different environment then you can run the
> script using the interpreter's direct specification.
> For example:
> /Users/kamilbregula/.virtualenvs/airflow/bin/python /path/to/script.py
>
> On Wed, Oct 16, 2019 at 2:06 AM Reed Villanueva <rv...@ucera.org>
> wrote:
> >
> > If I have multiple airflow dags with some overlapping python package
> dependencies, how can I keep each of these project deps. decoupled? Eg. if
> I had project A and B on same server I would run each of them with
> something like...
> >
> > source /path/to/virtualenv_a/activate
> > python script_a.py
> > deactivate
> > source /path/to/virtualenv_b/activate
> > python script_b.py
> > deactivate
> >
> > Basically, would like to run dags with the same situation (eg. each dag
> uses python scripts that have may have overlapping package deps. that I
> would like to develop separately (ie. not have to update all code using a
> package when want to update the package just for one project)). Note, I've
> been using the BashOperator to run python tasks like...
> >
> > do_stuff = BashOperator(
> > task_id='my_task',
> > bash_command='python /path/to/script.py'),
> > execution_timeout=timedelta(minutes=30),
> > dag=dag)
> >
> > Is there a way to get this working? Is there some other best-practice
> way that airflow intendeds for people to address (or avoid) these kinds of
> problems?
> >
> >
> > This electronic message is intended only for the named
> > recipient, and may contain information that is confidential or
> > privileged. If you are not the intended recipient, you are
> > hereby notified that any disclosure, copying, distribution or
> > use of the contents of this message is strictly prohibited. If
> > you have received this message in error or are not the named
> > recipient, please notify us immediately by contacting the
> > sender at the electronic mail address noted above, and delete
> > and destroy all copies of this message. Thank you.
>
--
This electronic message is intended only for the named
recipient, and may
contain information that is confidential or
privileged. If you are not the
intended recipient, you are
hereby notified that any disclosure, copying,
distribution or
use of the contents of this message is strictly
prohibited. If
you have received this message in error or are not the
named
recipient, please notify us immediately by contacting the
sender at
the electronic mail address noted above, and delete
and destroy all copies
of this message. Thank you.
Re: How to manage python packages between airflow dags?
Posted by Kamil Breguła <ka...@polidea.com>.
Don't use a wrapper, because only what it does changes the value of
the PATH variable.
11:16 $ type python
python is /Library/Frameworks/Python.framework/Versions/2.7/bin/python
11:16 $ workon airflow
11:16 $ type python
python is /Users/kamilbregula/.virtualenvs/airflow/bin/python
so if you want to use a different environment then you can run the
script using the interpreter's direct specification.
For example:
/Users/kamilbregula/.virtualenvs/airflow/bin/python /path/to/script.py
On Wed, Oct 16, 2019 at 2:06 AM Reed Villanueva <rv...@ucera.org> wrote:
>
> If I have multiple airflow dags with some overlapping python package dependencies, how can I keep each of these project deps. decoupled? Eg. if I had project A and B on same server I would run each of them with something like...
>
> source /path/to/virtualenv_a/activate
> python script_a.py
> deactivate
> source /path/to/virtualenv_b/activate
> python script_b.py
> deactivate
>
> Basically, would like to run dags with the same situation (eg. each dag uses python scripts that have may have overlapping package deps. that I would like to develop separately (ie. not have to update all code using a package when want to update the package just for one project)). Note, I've been using the BashOperator to run python tasks like...
>
> do_stuff = BashOperator(
> task_id='my_task',
> bash_command='python /path/to/script.py'),
> execution_timeout=timedelta(minutes=30),
> dag=dag)
>
> Is there a way to get this working? Is there some other best-practice way that airflow intendeds for people to address (or avoid) these kinds of problems?
>
>
> This electronic message is intended only for the named
> recipient, and may contain information that is confidential or
> privileged. If you are not the intended recipient, you are
> hereby notified that any disclosure, copying, distribution or
> use of the contents of this message is strictly prohibited. If
> you have received this message in error or are not the named
> recipient, please notify us immediately by contacting the
> sender at the electronic mail address noted above, and delete
> and destroy all copies of this message. Thank you.