You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Rajesh Chamarthi <ra...@gmail.com> on 2017/12/08 15:55:04 UTC
Environment variables - Oracle LD_LIBRARY_PATH
Hi all,
What is the right way to set environment variables to be used in DAGS.
Also, is it possible to export them such that currently running scheduler and worker processes can use it?
Context :
I am working on a DAG that involves making an Oracle connection, which requires the LD_LIBRARY_PATH variable to be set. I added it to /etc/profile.d of the airflow user. When I start a new shell + python virtualenv and run “airflow test dag_name task_name”, the task is able to use the path variable above and make the necessary connection.
However, when it runs through the scheduler, the task fails with the following error : “Oracle client library cannot be loaded”, presumably because it cannot read the LD_LIBRARY_PATH.
2) I have also tried the following code in my custom hook, to explicitly set the path. I see the same error.
if 'LD_LIBRARY_PATH' not in os.environ.keys():
os.environ['LD_LIBRARY_PATH']="/opt/oracle/instantclient_11_2"
logging.info("added ld library path", os.environ['LD_LIBRARY_PATH'])
Any pointers on how I can resolve this?
Thanks,
Rajesh
Re: Environment variables - Oracle LD_LIBRARY_PATH
Posted by Joy Gao <jo...@wepay.com>.
Hi Rajesh,
Odd that it still doesn't work after setting the worker's env profile. We
run LocalExecutor so this has never been an issue. I will leave to others
to comment.
(One thing worth trying is to explicitly export the variable `export
VAR=VAL` in a subprocess in the operator/hook so that it lives beyond the
scope of the python process.)
Cheers,
Joy
On Fri, Dec 8, 2017 at 4:32 PM, Rajesh Chamarthi <rajesh.chamarthi@gmail.com
> wrote:
> Hi Joy,
>
> Thank you, “airflow test” does not seem like a good way to test
> environment in that case :)
>
> Yes, I am running using celery executor and running the tasks in a
> distributed manner. I currently set the environment variable in the hook
> just before it tries to get a DB connection.
>
> We also use supervisord to manage the worker, scheduler and web server
> processes and I have changed the worker’s env profile to set the
> environment variable, but it does not work either.
>
>
> > On Dec 8, 2017, at 2:50 PM, Joy Gao <jo...@wepay.com> wrote:
> >
> > Hi Rajesh,
> >
> > `airflow test` executes the task synchronously in the same python process
> > (without touching the scheduler), which is probably why it went down the
> > happy path.
> >
> > Are you using celery executor and running the tasks in a distributed
> > manner? It* sounds like* the task execution env doesn't know about the
> > variable.
> >
> > Also depending on where you added the code to your custom hook, it may be
> > in a different process from which the task is executed, since os.environ
> > sets variable only for the scope/duration of the python process (and its
> > children).
> >
> >
> > On Fri, Dec 8, 2017 at 7:55 AM, Rajesh Chamarthi <
> rajesh.chamarthi@gmail.com
> >> wrote:
> >
> >> Hi all,
> >>
> >> What is the right way to set environment variables to be used in DAGS.
> >> Also, is it possible to export them such that currently running
> scheduler
> >> and worker processes can use it?
> >>
> >> Context :
> >>
> >> I am working on a DAG that involves making an Oracle connection, which
> >> requires the LD_LIBRARY_PATH variable to be set. I added it to
> >> /etc/profile.d of the airflow user. When I start a new shell + python
> >> virtualenv and run “airflow test dag_name task_name”, the task is able
> to
> >> use the path variable above and make the necessary connection.
> >>
> >> However, when it runs through the scheduler, the task fails with the
> >> following error : “Oracle client library cannot be loaded”, presumably
> >> because it cannot read the LD_LIBRARY_PATH.
> >>
> >> 2) I have also tried the following code in my custom hook, to explicitly
> >> set the path. I see the same error.
> >>
> >> if 'LD_LIBRARY_PATH' not in os.environ.keys():
> >> os.environ['LD_LIBRARY_PATH']="/opt/oracle/instantclient_11_2"
> >> logging.info("added ld library path", os.environ['LD_LIBRARY_PATH'])
> >>
> >> Any pointers on how I can resolve this?
> >>
> >> Thanks,
> >> Rajesh
> >>
> >>
>
>
Re: Environment variables - Oracle LD_LIBRARY_PATH
Posted by Rajesh Chamarthi <ra...@gmail.com>.
Will give this a try. I tried the airflow worker’s conf and it did d not
work.
I fixed my problem (temporarily) by setting the environment variable in the
python __init__ code before it calls the database connection create
statement. (Using python os.environ).
On Mon, Dec 11, 2017 at 9:50 AM William Pursell <wi...@wepay.com> wrote:
> It's perhaps not the cleanest solution, but I have had success setting the
> environment in the supervisord config file. Eg: add the line
>
> environment=LD_LIBRARY_PATH="/opt/oracle/instantclient_11_2"
>
> On Fri, Dec 8, 2017 at 4:32 PM, Rajesh Chamarthi <
> rajesh.chamarthi@gmail.com
> > wrote:
>
> > Hi Joy,
> >
> > Thank you, “airflow test” does not seem like a good way to test
> > environment in that case :)
> >
> > Yes, I am running using celery executor and running the tasks in a
> > distributed manner. I currently set the environment variable in the hook
> > just before it tries to get a DB connection.
> >
> > We also use supervisord to manage the worker, scheduler and web server
> > processes and I have changed the worker’s env profile to set the
> > environment variable, but it does not work either.
> >
> >
> > > On Dec 8, 2017, at 2:50 PM, Joy Gao <jo...@wepay.com> wrote:
> > >
> > > Hi Rajesh,
> > >
> > > `airflow test` executes the task synchronously in the same python
> process
> > > (without touching the scheduler), which is probably why it went down
> the
> > > happy path.
> > >
> > > Are you using celery executor and running the tasks in a distributed
> > > manner? It* sounds like* the task execution env doesn't know about the
> > > variable.
> > >
> > > Also depending on where you added the code to your custom hook, it may
> be
> > > in a different process from which the task is executed, since
> os.environ
> > > sets variable only for the scope/duration of the python process (and
> its
> > > children).
> > >
> > >
> > > On Fri, Dec 8, 2017 at 7:55 AM, Rajesh Chamarthi <
> > rajesh.chamarthi@gmail.com
> > >> wrote:
> > >
> > >> Hi all,
> > >>
> > >> What is the right way to set environment variables to be used in DAGS.
> > >> Also, is it possible to export them such that currently running
> > scheduler
> > >> and worker processes can use it?
> > >>
> > >> Context :
> > >>
> > >> I am working on a DAG that involves making an Oracle connection, which
> > >> requires the LD_LIBRARY_PATH variable to be set. I added it to
> > >> /etc/profile.d of the airflow user. When I start a new shell + python
> > >> virtualenv and run “airflow test dag_name task_name”, the task is able
> > to
> > >> use the path variable above and make the necessary connection.
> > >>
> > >> However, when it runs through the scheduler, the task fails with the
> > >> following error : “Oracle client library cannot be loaded”, presumably
> > >> because it cannot read the LD_LIBRARY_PATH.
> > >>
> > >> 2) I have also tried the following code in my custom hook, to
> explicitly
> > >> set the path. I see the same error.
> > >>
> > >> if 'LD_LIBRARY_PATH' not in os.environ.keys():
> > >> os.environ['LD_LIBRARY_PATH']="/opt/oracle/instantclient_11_2"
> > >> logging.info("added ld library path",
> os.environ['LD_LIBRARY_PATH'])
> > >>
> > >> Any pointers on how I can resolve this?
> > >>
> > >> Thanks,
> > >> Rajesh
> > >>
> > >>
> >
> >
>
Re: Environment variables - Oracle LD_LIBRARY_PATH
Posted by William Pursell <wi...@wepay.com>.
It's perhaps not the cleanest solution, but I have had success setting the
environment in the supervisord config file. Eg: add the line
environment=LD_LIBRARY_PATH="/opt/oracle/instantclient_11_2"
On Fri, Dec 8, 2017 at 4:32 PM, Rajesh Chamarthi <rajesh.chamarthi@gmail.com
> wrote:
> Hi Joy,
>
> Thank you, “airflow test” does not seem like a good way to test
> environment in that case :)
>
> Yes, I am running using celery executor and running the tasks in a
> distributed manner. I currently set the environment variable in the hook
> just before it tries to get a DB connection.
>
> We also use supervisord to manage the worker, scheduler and web server
> processes and I have changed the worker’s env profile to set the
> environment variable, but it does not work either.
>
>
> > On Dec 8, 2017, at 2:50 PM, Joy Gao <jo...@wepay.com> wrote:
> >
> > Hi Rajesh,
> >
> > `airflow test` executes the task synchronously in the same python process
> > (without touching the scheduler), which is probably why it went down the
> > happy path.
> >
> > Are you using celery executor and running the tasks in a distributed
> > manner? It* sounds like* the task execution env doesn't know about the
> > variable.
> >
> > Also depending on where you added the code to your custom hook, it may be
> > in a different process from which the task is executed, since os.environ
> > sets variable only for the scope/duration of the python process (and its
> > children).
> >
> >
> > On Fri, Dec 8, 2017 at 7:55 AM, Rajesh Chamarthi <
> rajesh.chamarthi@gmail.com
> >> wrote:
> >
> >> Hi all,
> >>
> >> What is the right way to set environment variables to be used in DAGS.
> >> Also, is it possible to export them such that currently running
> scheduler
> >> and worker processes can use it?
> >>
> >> Context :
> >>
> >> I am working on a DAG that involves making an Oracle connection, which
> >> requires the LD_LIBRARY_PATH variable to be set. I added it to
> >> /etc/profile.d of the airflow user. When I start a new shell + python
> >> virtualenv and run “airflow test dag_name task_name”, the task is able
> to
> >> use the path variable above and make the necessary connection.
> >>
> >> However, when it runs through the scheduler, the task fails with the
> >> following error : “Oracle client library cannot be loaded”, presumably
> >> because it cannot read the LD_LIBRARY_PATH.
> >>
> >> 2) I have also tried the following code in my custom hook, to explicitly
> >> set the path. I see the same error.
> >>
> >> if 'LD_LIBRARY_PATH' not in os.environ.keys():
> >> os.environ['LD_LIBRARY_PATH']="/opt/oracle/instantclient_11_2"
> >> logging.info("added ld library path", os.environ['LD_LIBRARY_PATH'])
> >>
> >> Any pointers on how I can resolve this?
> >>
> >> Thanks,
> >> Rajesh
> >>
> >>
>
>
Re: Environment variables - Oracle LD_LIBRARY_PATH
Posted by Rajesh Chamarthi <ra...@gmail.com>.
Hi Joy,
Thank you, “airflow test” does not seem like a good way to test environment in that case :)
Yes, I am running using celery executor and running the tasks in a distributed manner. I currently set the environment variable in the hook just before it tries to get a DB connection.
We also use supervisord to manage the worker, scheduler and web server processes and I have changed the worker’s env profile to set the environment variable, but it does not work either.
> On Dec 8, 2017, at 2:50 PM, Joy Gao <jo...@wepay.com> wrote:
>
> Hi Rajesh,
>
> `airflow test` executes the task synchronously in the same python process
> (without touching the scheduler), which is probably why it went down the
> happy path.
>
> Are you using celery executor and running the tasks in a distributed
> manner? It* sounds like* the task execution env doesn't know about the
> variable.
>
> Also depending on where you added the code to your custom hook, it may be
> in a different process from which the task is executed, since os.environ
> sets variable only for the scope/duration of the python process (and its
> children).
>
>
> On Fri, Dec 8, 2017 at 7:55 AM, Rajesh Chamarthi <rajesh.chamarthi@gmail.com
>> wrote:
>
>> Hi all,
>>
>> What is the right way to set environment variables to be used in DAGS.
>> Also, is it possible to export them such that currently running scheduler
>> and worker processes can use it?
>>
>> Context :
>>
>> I am working on a DAG that involves making an Oracle connection, which
>> requires the LD_LIBRARY_PATH variable to be set. I added it to
>> /etc/profile.d of the airflow user. When I start a new shell + python
>> virtualenv and run “airflow test dag_name task_name”, the task is able to
>> use the path variable above and make the necessary connection.
>>
>> However, when it runs through the scheduler, the task fails with the
>> following error : “Oracle client library cannot be loaded”, presumably
>> because it cannot read the LD_LIBRARY_PATH.
>>
>> 2) I have also tried the following code in my custom hook, to explicitly
>> set the path. I see the same error.
>>
>> if 'LD_LIBRARY_PATH' not in os.environ.keys():
>> os.environ['LD_LIBRARY_PATH']="/opt/oracle/instantclient_11_2"
>> logging.info("added ld library path", os.environ['LD_LIBRARY_PATH'])
>>
>> Any pointers on how I can resolve this?
>>
>> Thanks,
>> Rajesh
>>
>>
Re: Environment variables - Oracle LD_LIBRARY_PATH
Posted by Joy Gao <jo...@wepay.com>.
Hi Rajesh,
`airflow test` executes the task synchronously in the same python process
(without touching the scheduler), which is probably why it went down the
happy path.
Are you using celery executor and running the tasks in a distributed
manner? It* sounds like* the task execution env doesn't know about the
variable.
Also depending on where you added the code to your custom hook, it may be
in a different process from which the task is executed, since os.environ
sets variable only for the scope/duration of the python process (and its
children).
On Fri, Dec 8, 2017 at 7:55 AM, Rajesh Chamarthi <rajesh.chamarthi@gmail.com
> wrote:
> Hi all,
>
> What is the right way to set environment variables to be used in DAGS.
> Also, is it possible to export them such that currently running scheduler
> and worker processes can use it?
>
> Context :
>
> I am working on a DAG that involves making an Oracle connection, which
> requires the LD_LIBRARY_PATH variable to be set. I added it to
> /etc/profile.d of the airflow user. When I start a new shell + python
> virtualenv and run “airflow test dag_name task_name”, the task is able to
> use the path variable above and make the necessary connection.
>
> However, when it runs through the scheduler, the task fails with the
> following error : “Oracle client library cannot be loaded”, presumably
> because it cannot read the LD_LIBRARY_PATH.
>
> 2) I have also tried the following code in my custom hook, to explicitly
> set the path. I see the same error.
>
> if 'LD_LIBRARY_PATH' not in os.environ.keys():
> os.environ['LD_LIBRARY_PATH']="/opt/oracle/instantclient_11_2"
> logging.info("added ld library path", os.environ['LD_LIBRARY_PATH'])
>
> Any pointers on how I can resolve this?
>
> Thanks,
> Rajesh
>
>