You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Frank Maritato <fm...@opentable.com> on 2017/10/16 21:58:47 UTC
programatically configure aws_default, emr_default connections?
Hi,
Is there any way to programatically configure the aws_default, emr_default or other airflow connections? The reason I'm asking is that I'm creating a docker container and I want it to be preconfigured without ui interaction. I looked at Puckel's docker-airflow project and I didn’t see anything like this being done in there.
Thanks!
--
Frank Maritato
Re: programatically configure aws_default, emr_default connections?
Posted by Ash Berlin-Taylor <as...@firemirror.com>.
It is possible to configure connections via environment variables - AIRFLOW_CONN_*, for instance
`AIRFLOW_CONN_BRAND_INDEX_DB=postgresql+psycopg2://airflow:airflow@db/brand_index` <postgresql+psycopg2://airflow:airflow@db/brand_index%60>
Will be used when a connection id of "brand_index_db" (of any case) is loaded via a hook: https://github.com/apache/incubator-airflow/blob/master/airflow/hooks/base_hook.py#L58-L73 <https://github.com/apache/incubator-airflow/blob/master/airflow/hooks/base_hook.py#L58-L73>
Sadly this doesn't let us set the extra field https://github.com/apache/incubator-airflow/blob/15feb7dd3f39ba7926ae5817d488e4e54a3d7742/airflow/models.py#L577-L590 <https://github.com/apache/incubator-airflow/blob/15feb7dd3f39ba7926ae5817d488e4e54a3d7742/airflow/models.py#L577-L590> and it has some problems trying to parse uri schemas for things like `google_cloud_platform://`
But it might help you in some cases.
-ash
> On 16 Oct 2017, at 22:58, Frank Maritato <fmaritato@opentable.com <ma...@opentable.com>> wrote:
>
> Hi,
>
> Is there any way to programatically configure the aws_default, emr_default or other airflow connections? The reason I'm asking is that I'm creating a docker container and I want it to be preconfigured without ui interaction. I looked at Puckel's docker-airflow project and I didn’t see anything like this being done in there.
>
> Thanks!
> --
> Frank Maritato
>
>
Re: programatically configure aws_default, emr_default connections?
Posted by Andrew Phillips <an...@apache.org>.
> Is there any way to programatically configure the aws_default,
> emr_default or other airflow connections?
In a pre-1.8.x installation we used the following script:
--- snip ---
#!/usr/bin/python
import json
from airflow import models
from airflow.utils import db as db_utils
CONNECTIONS = [
models.Connection(
conn_id='foo', conn_type='http',
host='...', port='8080'),
models.Connection(
conn_id='bar', conn_type='ssh',
host='...'),
models.Connection(
conn_id='baz', conn_type='http',
host='...', port='7099'),
models.Connection(
conn_id='sqs_us_east_1', conn_type='s3',
extra=json.dumps({ 'region': 'us-east-1' }))
]
for connection in CONNECTIONS:
print "Adding connection {}".format(connection.conn_id)
db_utils.merge_conn(connection)
--- snip ---
This was for adding *new* connections, but I would suspect that this or
similar code could also be used to modify existing connections.
There may well be a better approach for Airflow 1.8.x, though.
Regards
ap
Re: programatically configure aws_default, emr_default connections?
Posted by Joy Gao <jo...@wepay.com>.
Just want to add that a PR (
https://github.com/apache/incubator-airflow/pull/2525) is underway to allow
adding connections via CLI (bypassing the uri schema issue).
On Wed, Oct 18, 2017 at 11:45 AM, Frank Maritato <fm...@opentable.com>
wrote:
> Thanks, Andrew! This works nicely.
>
> On 10/16/17, 3:01 PM, "Andrew Chen" <an...@databricks.com> wrote:
>
> `airflow connections --add` might be what you're looking for.
>
> On Mon, Oct 16, 2017 at 2:58 PM, Frank Maritato <
> fmaritato@opentable.com>
> wrote:
>
> > Hi,
> >
> > Is there any way to programatically configure the aws_default,
> emr_default
> > or other airflow connections? The reason I'm asking is that I'm
> creating a
> > docker container and I want it to be preconfigured without ui
> interaction.
> > I looked at Puckel's docker-airflow project and I didn’t see
> anything like
> > this being done in there.
> >
> > Thanks!
> > --
> > Frank Maritato
> >
> >
> >
>
>
>
Re: programatically configure aws_default, emr_default connections?
Posted by Frank Maritato <fm...@opentable.com>.
Thanks, Andrew! This works nicely.
On 10/16/17, 3:01 PM, "Andrew Chen" <an...@databricks.com> wrote:
`airflow connections --add` might be what you're looking for.
On Mon, Oct 16, 2017 at 2:58 PM, Frank Maritato <fm...@opentable.com>
wrote:
> Hi,
>
> Is there any way to programatically configure the aws_default, emr_default
> or other airflow connections? The reason I'm asking is that I'm creating a
> docker container and I want it to be preconfigured without ui interaction.
> I looked at Puckel's docker-airflow project and I didn’t see anything like
> this being done in there.
>
> Thanks!
> --
> Frank Maritato
>
>
>
Re: programatically configure aws_default, emr_default connections?
Posted by Andrew Chen <an...@databricks.com>.
`airflow connections --add` might be what you're looking for.
On Mon, Oct 16, 2017 at 2:58 PM, Frank Maritato <fm...@opentable.com>
wrote:
> Hi,
>
> Is there any way to programatically configure the aws_default, emr_default
> or other airflow connections? The reason I'm asking is that I'm creating a
> docker container and I want it to be preconfigured without ui interaction.
> I looked at Puckel's docker-airflow project and I didn’t see anything like
> this being done in there.
>
> Thanks!
> --
> Frank Maritato
>
>
>
Re: programatically configure aws_default, emr_default connections?
Posted by Andy Hadjigeorgiou <an...@gmail.com>.
The connections are just db rows - is it possible to run a couple sql queries on the database connected to airflow when building the container to create them? (If you also want to avoid cli)
> On Oct 16, 2017, at 5:58 PM, Frank Maritato <fm...@opentable.com> wrote:
>
> Hi,
>
> Is there any way to programatically configure the aws_default, emr_default or other airflow connections? The reason I'm asking is that I'm creating a docker container and I want it to be preconfigured without ui interaction. I looked at Puckel's docker-airflow project and I didn’t see anything like this being done in there.
>
> Thanks!
> --
> Frank Maritato
>
>