You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Somasundaram Sekar <so...@tigeranalytics.com> on 2017/11/09 07:30:31 UTC
Airflow configuration in environment variable not working
I tried using ENV variable to configure connection urls, I have an ami that
is preconfigured with alchemy_conn and broker_url etc, I have written
environment variables to /etc/environment in the instances being spun up
from the amis to override the properties in airflow.cfg file. I was able to
access the variables from python code as well.
But it doesn't seem to take effect while running ariflow, tried restarting
the process as well but didn't work, it still points to the one in
airflow.cfg file
Any help appreciated.
Regards,
Somasundaram S
Re: Airflow configuration in environment variable not working
Posted by Ash Berlin-Taylor <as...@firemirror.com>.
Connections are configured differently: http://airflow.readthedocs.io/en/latest/concepts.html#connections
> When referencing the connection in the Airflow pipeline, the conn_id should be the name of the variable without the prefix. For example, if the `conn_id` is named `postgres_master` the environment variable should be named `AIRFLOW_CONN_POSTGRES_MASTER` (note that the environment variable must be all uppercase). Airflow assumes the value returned from the environment variable to be in a URI format (e.g. `postgres://user:password@localhost:5432/master or s3://accesskey:secretkey@S3`).
If your sql alchemy conn isn't being picked up by airflow it is possible that /etc/environment isn't affecting running services, just interactive logins/shells. If you are using systemd add EnvironmentFile=/etc/environment (I think that's the right stanza) to make it load that too
-ash
> On 9 Nov 2017, at 08:35, Somasundaram Sekar <so...@tigeranalytics.com> wrote:
>
> Below are the env variables configured
>
> AIRFLOW__CORE__SQL_ALCHEMY_CONN=mysql://XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
> AIRFLOW__CELERY__BROKER_URL=redis://XXXXXXXXXXXXXXXXXXXX/1
> AIRFLOW__CELERY__CELERY_RESULT_BACKEND=db+mysql://XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXx
> AIRFLOW__WEBSERVER__AUTHENTICATE=True
> AIRFLOW__WEBSERVER__AUTH_BACKEND=airflow.contrib.auth.backends.google_auth
> AIRFLOW__GOOGLE__CLIENT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXx
> AIRFLOW__GOOGLE__CLIENT_SECRET=XXXXXXXXXXXXXXXXXXxx
> AIRFLOW__GOOGLE__OAUTH_CALLBACK_ROUTE=/
> AIRFLOW__GOOGLE__DOMAIN=XXXXXXXXXXXXXXXXX
>
> -----Original Message-----
> From: Gerard Toonstra [mailto:gtoonstra@gmail.com]
> Sent: Thursday, November 9, 2017 1:59 PM
> To: dev@airflow.incubator.apache.org
> Subject: Re: Airflow configuration in environment variable not working
>
> What's the variable key you are using. Does it follow this convention?
>
> https://airflow.apache.org/configuration.html
>
> That's AIRFLOW (two underscores) configuration section (two underscores)
> env var.
>
>
> G>
>
>
> On Thu, Nov 9, 2017 at 8:30 AM, Somasundaram Sekar <
> somasundar.sekar@tigeranalytics.com> wrote:
>
>> I tried using ENV variable to configure connection urls, I have an ami
>> that is preconfigured with alchemy_conn and broker_url etc, I have
>> written environment variables to /etc/environment in the instances
>> being spun up from the amis to override the properties in airflow.cfg
>> file. I was able to access the variables from python code as well.
>>
>> But it doesn't seem to take effect while running ariflow, tried
>> restarting the process as well but didn't work, it still points to the
>> one in airflow.cfg file
>>
>> Any help appreciated.
>>
>> Regards,
>>
>> Somasundaram S
>>
Re: Airflow configuration in environment variable not working
Posted by Gerard Toonstra <gt...@gmail.com>.
Idea to check:
If you're running airflow as a configured service (systemd or what), then
it's possible the /etc/environment is not
picked up, because it's not an interactive session.
You'd need to explicitly set the env vars you'd like to load in like this:
[Service]
EnvironmentFile=-/etc/environment
You can try running airflow from an interactive console first to see if
that fixes it, then run it as a service.
Without the minus, the systemd service fails to start if the file is
missing, which may be what you prefer.
On Thu, Nov 9, 2017 at 9:35 AM, Somasundaram Sekar <
somasundar.sekar@tigeranalytics.com> wrote:
> Below are the env variables configured
>
> AIRFLOW__CORE__SQL_ALCHEMY_CONN=mysql://XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
> AIRFLOW__CELERY__BROKER_URL=redis://XXXXXXXXXXXXXXXXXXXX/1
> AIRFLOW__CELERY__CELERY_RESULT_BACKEND=db+mysql://
> XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXx
> AIRFLOW__WEBSERVER__AUTHENTICATE=True
> AIRFLOW__WEBSERVER__AUTH_BACKEND=airflow.contrib.auth.backends.google_auth
> AIRFLOW__GOOGLE__CLIENT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXx
> AIRFLOW__GOOGLE__CLIENT_SECRET=XXXXXXXXXXXXXXXXXXxx
> AIRFLOW__GOOGLE__OAUTH_CALLBACK_ROUTE=/
> AIRFLOW__GOOGLE__DOMAIN=XXXXXXXXXXXXXXXXX
>
> -----Original Message-----
> From: Gerard Toonstra [mailto:gtoonstra@gmail.com]
> Sent: Thursday, November 9, 2017 1:59 PM
> To: dev@airflow.incubator.apache.org
> Subject: Re: Airflow configuration in environment variable not working
>
> What's the variable key you are using. Does it follow this convention?
>
> https://airflow.apache.org/configuration.html
>
> That's AIRFLOW (two underscores) configuration section (two underscores)
> env var.
>
>
> G>
>
>
> On Thu, Nov 9, 2017 at 8:30 AM, Somasundaram Sekar <
> somasundar.sekar@tigeranalytics.com> wrote:
>
> > I tried using ENV variable to configure connection urls, I have an ami
> > that is preconfigured with alchemy_conn and broker_url etc, I have
> > written environment variables to /etc/environment in the instances
> > being spun up from the amis to override the properties in airflow.cfg
> > file. I was able to access the variables from python code as well.
> >
> > But it doesn't seem to take effect while running ariflow, tried
> > restarting the process as well but didn't work, it still points to the
> > one in airflow.cfg file
> >
> > Any help appreciated.
> >
> > Regards,
> >
> > Somasundaram S
> >
>
RE: Airflow configuration in environment variable not working
Posted by Somasundaram Sekar <so...@tigeranalytics.com>.
Below are the env variables configured
AIRFLOW__CORE__SQL_ALCHEMY_CONN=mysql://XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
AIRFLOW__CELERY__BROKER_URL=redis://XXXXXXXXXXXXXXXXXXXX/1
AIRFLOW__CELERY__CELERY_RESULT_BACKEND=db+mysql://XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXx
AIRFLOW__WEBSERVER__AUTHENTICATE=True
AIRFLOW__WEBSERVER__AUTH_BACKEND=airflow.contrib.auth.backends.google_auth
AIRFLOW__GOOGLE__CLIENT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXx
AIRFLOW__GOOGLE__CLIENT_SECRET=XXXXXXXXXXXXXXXXXXxx
AIRFLOW__GOOGLE__OAUTH_CALLBACK_ROUTE=/
AIRFLOW__GOOGLE__DOMAIN=XXXXXXXXXXXXXXXXX
-----Original Message-----
From: Gerard Toonstra [mailto:gtoonstra@gmail.com]
Sent: Thursday, November 9, 2017 1:59 PM
To: dev@airflow.incubator.apache.org
Subject: Re: Airflow configuration in environment variable not working
What's the variable key you are using. Does it follow this convention?
https://airflow.apache.org/configuration.html
That's AIRFLOW (two underscores) configuration section (two underscores)
env var.
G>
On Thu, Nov 9, 2017 at 8:30 AM, Somasundaram Sekar <
somasundar.sekar@tigeranalytics.com> wrote:
> I tried using ENV variable to configure connection urls, I have an ami
> that is preconfigured with alchemy_conn and broker_url etc, I have
> written environment variables to /etc/environment in the instances
> being spun up from the amis to override the properties in airflow.cfg
> file. I was able to access the variables from python code as well.
>
> But it doesn't seem to take effect while running ariflow, tried
> restarting the process as well but didn't work, it still points to the
> one in airflow.cfg file
>
> Any help appreciated.
>
> Regards,
>
> Somasundaram S
>
Re: Airflow configuration in environment variable not working
Posted by Gerard Toonstra <gt...@gmail.com>.
What's the variable key you are using. Does it follow this convention?
https://airflow.apache.org/configuration.html
That's AIRFLOW (two underscores) configuration section (two underscores)
env var.
G>
On Thu, Nov 9, 2017 at 8:30 AM, Somasundaram Sekar <
somasundar.sekar@tigeranalytics.com> wrote:
> I tried using ENV variable to configure connection urls, I have an ami that
> is preconfigured with alchemy_conn and broker_url etc, I have written
> environment variables to /etc/environment in the instances being spun up
> from the amis to override the properties in airflow.cfg file. I was able to
> access the variables from python code as well.
>
> But it doesn't seem to take effect while running ariflow, tried restarting
> the process as well but didn't work, it still points to the one in
> airflow.cfg file
>
> Any help appreciated.
>
> Regards,
>
> Somasundaram S
>