You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Nicholas Hodgkinson <ni...@collectivehealth.com> on 2017/03/07 21:49:53 UTC
Adding Variables and Connections via script
I would like to be able to create a script to assist local development
which would populate several Connections and Variables that are used across
our organization; is there a way that I can add those from the command line
or Python script without having to manually enter them via the UI?
Thanks,
-Nik
nik.hodgkinson@collectivehealth.com
--
Read our founder's story.
<https://collectivehealth.com/blog/started-collective-health/>
*This message may contain confidential, proprietary, or protected
information. If you are not the intended recipient, you may not review,
copy, or distribute this message. If you received this message in error,
please notify the sender by reply email and delete this message.*
Re: Adding Variables and Connections via script
Posted by Ali Uz <al...@gmail.com>.
Hello Nicholas,
yes there is:
for variables you can do:
from airflow.models import Variable
Variable.set(key, value)
for connections you can do:
from airflow import settings
from airflow.models import Connection
def create_connection(
conn_id, conn_type, host, schema, login, password, port=None,
extra=None):
logging.info('Creating connection %s' % conn_id)
new_conn = Connection()
new_conn.conn_id = conn_id
new_conn.conn_type = conn_type
new_conn.host = host
new_conn.schema = schema
new_conn.login = login
new_conn.password = password
if port:
new_conn.port = port
if extra:
new_conn.extra = extra
session = settings.Session()
session.add(new_conn)
session.commit()
session.close()
You can extend and modify these to fit your needs as necessary.
On Tue, Mar 7, 2017 at 11:49 PM, Nicholas Hodgkinson <
nik.hodgkinson@collectivehealth.com> wrote:
> I would like to be able to create a script to assist local development
> which would populate several Connections and Variables that are used across
> our organization; is there a way that I can add those from the command line
> or Python script without having to manually enter them via the UI?
>
> Thanks,
> -Nik
> nik.hodgkinson@collectivehealth.com
>
> --
>
>
> Read our founder's story.
> <https://collectivehealth.com/blog/started-collective-health/>
>
> *This message may contain confidential, proprietary, or protected
> information. If you are not the intended recipient, you may not review,
> copy, or distribute this message. If you received this message in error,
> please notify the sender by reply email and delete this message.*
>
Re: Adding Variables and Connections via script
Posted by siddharth anand <sa...@apache.org>.
Yes.. I recall that PR and I also recall adding support to manage variables
via the CLI a long time back.. both of those should be in the 1.8 release.
-s
On Wed, Mar 8, 2017 at 10:18 AM, Nicholas Hodgkinson <
nik.hodgkinson@collectivehealth.com> wrote:
> Thanks everyone! This is super helpful!
>
> -Nik
> nik.hodgkinson@collectivehealth.com
>
>
> On Tue, Mar 7, 2017 at 2:23 PM, Boris Tyukin <bo...@boristyukin.com>
> wrote:
>
> > To add to Ali's reply, there was a PR for connections and cli
> > https://github.com/apache/incubator-airflow/pull/1802
> >
> > hopefully it will make to 1.8
> >
> >
> > On Tue, Mar 7, 2017 at 4:49 PM, Nicholas Hodgkinson <
> > nik.hodgkinson@collectivehealth.com> wrote:
> >
> > > I would like to be able to create a script to assist local development
> > > which would populate several Connections and Variables that are used
> > across
> > > our organization; is there a way that I can add those from the command
> > line
> > > or Python script without having to manually enter them via the UI?
> > >
> > > Thanks,
> > > -Nik
> > > nik.hodgkinson@collectivehealth.com
> > >
> > > --
> > >
> > >
> > > Read our founder's story.
> > > <https://collectivehealth.com/blog/started-collective-health/>
> > >
> > > *This message may contain confidential, proprietary, or protected
> > > information. If you are not the intended recipient, you may not
> review,
> > > copy, or distribute this message. If you received this message in
> error,
> > > please notify the sender by reply email and delete this message.*
> > >
> >
>
> --
>
>
> Read our founder's story.
> <https://collectivehealth.com/blog/started-collective-health/>
>
> *This message may contain confidential, proprietary, or protected
> information. If you are not the intended recipient, you may not review,
> copy, or distribute this message. If you received this message in error,
> please notify the sender by reply email and delete this message.*
>
Re: Adding Variables and Connections via script
Posted by Nicholas Hodgkinson <ni...@collectivehealth.com>.
Thanks everyone! This is super helpful!
-Nik
nik.hodgkinson@collectivehealth.com
On Tue, Mar 7, 2017 at 2:23 PM, Boris Tyukin <bo...@boristyukin.com> wrote:
> To add to Ali's reply, there was a PR for connections and cli
> https://github.com/apache/incubator-airflow/pull/1802
>
> hopefully it will make to 1.8
>
>
> On Tue, Mar 7, 2017 at 4:49 PM, Nicholas Hodgkinson <
> nik.hodgkinson@collectivehealth.com> wrote:
>
> > I would like to be able to create a script to assist local development
> > which would populate several Connections and Variables that are used
> across
> > our organization; is there a way that I can add those from the command
> line
> > or Python script without having to manually enter them via the UI?
> >
> > Thanks,
> > -Nik
> > nik.hodgkinson@collectivehealth.com
> >
> > --
> >
> >
> > Read our founder's story.
> > <https://collectivehealth.com/blog/started-collective-health/>
> >
> > *This message may contain confidential, proprietary, or protected
> > information. If you are not the intended recipient, you may not review,
> > copy, or distribute this message. If you received this message in error,
> > please notify the sender by reply email and delete this message.*
> >
>
--
Read our founder's story.
<https://collectivehealth.com/blog/started-collective-health/>
*This message may contain confidential, proprietary, or protected
information. If you are not the intended recipient, you may not review,
copy, or distribute this message. If you received this message in error,
please notify the sender by reply email and delete this message.*
Re: Adding Variables and Connections via script
Posted by Boris Tyukin <bo...@boristyukin.com>.
To add to Ali's reply, there was a PR for connections and cli
https://github.com/apache/incubator-airflow/pull/1802
hopefully it will make to 1.8
On Tue, Mar 7, 2017 at 4:49 PM, Nicholas Hodgkinson <
nik.hodgkinson@collectivehealth.com> wrote:
> I would like to be able to create a script to assist local development
> which would populate several Connections and Variables that are used across
> our organization; is there a way that I can add those from the command line
> or Python script without having to manually enter them via the UI?
>
> Thanks,
> -Nik
> nik.hodgkinson@collectivehealth.com
>
> --
>
>
> Read our founder's story.
> <https://collectivehealth.com/blog/started-collective-health/>
>
> *This message may contain confidential, proprietary, or protected
> information. If you are not the intended recipient, you may not review,
> copy, or distribute this message. If you received this message in error,
> please notify the sender by reply email and delete this message.*
>
Re: Adding Variables and Connections via script
Posted by Daniel Huang <dx...@gmail.com>.
Yep, look at airflow connections and airflow variables to perform CRUD
operations via command line. For variables, it also supports importing from
a JSON file, which is pretty handy.
Alternatively, connections can be defined with environment variables, see
last paragraph of
https://airflow.incubator.apache.org/concepts.html#connections. Also,
variables can be set in code with airflow.models.Variable.set().
On Tue, Mar 7, 2017 at 1:49 PM, Nicholas Hodgkinson <
nik.hodgkinson@collectivehealth.com> wrote:
> I would like to be able to create a script to assist local development
> which would populate several Connections and Variables that are used across
> our organization; is there a way that I can add those from the command line
> or Python script without having to manually enter them via the UI?
>
> Thanks,
> -Nik
> nik.hodgkinson@collectivehealth.com
>
> --
>
>
> Read our founder's story.
> <https://collectivehealth.com/blog/started-collective-health/>
>
> *This message may contain confidential, proprietary, or protected
> information. If you are not the intended recipient, you may not review,
> copy, or distribute this message. If you received this message in error,
> please notify the sender by reply email and delete this message.*
>