You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by הילה ויזן <hi...@gmail.com> on 2016/09/05 04:40:33 UTC
creating a pool automatically
Hi,
we use a pool of size 1 to restrict parallelism of a task in dag.
We created the pool via airflow webserver.
Is there any other way to create it via a script? maybe sql query on DB?
I didn't find an appropriate CLI command.
Thanks,
Hila
Re: creating a pool automatically
Posted by siddharth anand <sa...@apache.org>.
As with other Airflow CLI commands, you can pass the *-h* flag to see
usage.
[image: Inline image 2]
To expose the list of available airflow CLI commands, type *airflow*
or *airflow
-h*
Not sure when this will be released, it won't hurt to update your fork to
this commit if releasing to your QA/PROD envs from your own fork, which I
recommend, so that you don't need to be tied to our release schedule.
-s
Re: creating a pool automatically
Posted by הילה ויזן <hi...@gmail.com>.
when are you expecting release version to include this fix?
On Thu, Sep 8, 2016 at 5:02 PM, הילה ויזן <hi...@gmail.com> wrote:
> Hi,
> can you also provide command line parameters?
>
> i didn't find it in the documentation
>
>
> On Wed, Sep 7, 2016 at 2:50 AM, siddharth anand <sa...@apache.org> wrote:
>
>> You can now create/modify pools via the Airflow CLI.
>>
>> https://github.com/apache/incubator-airflow/pull/1735/commits
>>
>> -s
>>
>> Resources that are typically modified via the webserver, such as pools,
>> connections, & variables, should eventually be managed via the CLI to
>> allow
>> for better integration with automation used to provision systems or deploy
>> new dags. Variables and pools (and I believe connections) are all done via
>> the CLI.
>>
>>
>> On Tue, Sep 6, 2016 at 12:22 PM, harish singh <ha...@gmail.com>
>> wrote:
>>
>> > This is how we have been doing it.
>> >
>> > pool_name = "pool_A"
>> >
>> > num_slots = 10
>> > # creating pool
>> > session = airflow.settings.Session()
>> > pool = (
>> > session.query(Pool)
>> > .filter(Pool.pool == pool_name)
>> > .first())
>> > if not pool:
>> > logging.info("Creating pool: {} with slots:
>> > ".format(pool_name, str(num_slots)))
>> > session.add(Pool(pool=pool_name, slots=num_slots))
>> > session.commit()
>> > session.close()
>> >
>> >
>> >
>> > On Tue, Sep 6, 2016 at 12:18 PM, Lance Norskog <lance.norskog@gmail.com
>> >
>> > wrote:
>> >
>> > > In the web UI, Admin->Pools lets you create, edit & view all Pools.
>> > >
>> > > On Sun, Sep 4, 2016 at 9:40 PM, הילה ויזן <hi...@gmail.com> wrote:
>> > >
>> > > > Hi,
>> > > > we use a pool of size 1 to restrict parallelism of a task in dag.
>> > > > We created the pool via airflow webserver.
>> > > >
>> > > > Is there any other way to create it via a script? maybe sql query on
>> > DB?
>> > > > I didn't find an appropriate CLI command.
>> > > >
>> > > > Thanks,
>> > > > Hila
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Lance Norskog
>> > > lance.norskog@gmail.com
>> > > Redwood City, CA
>> > >
>> >
>>
>
>
Re: creating a pool automatically
Posted by הילה ויזן <hi...@gmail.com>.
Hi,
can you also provide command line parameters?
i didn't find it in the documentation
On Wed, Sep 7, 2016 at 2:50 AM, siddharth anand <sa...@apache.org> wrote:
> You can now create/modify pools via the Airflow CLI.
>
> https://github.com/apache/incubator-airflow/pull/1735/commits
>
> -s
>
> Resources that are typically modified via the webserver, such as pools,
> connections, & variables, should eventually be managed via the CLI to allow
> for better integration with automation used to provision systems or deploy
> new dags. Variables and pools (and I believe connections) are all done via
> the CLI.
>
>
> On Tue, Sep 6, 2016 at 12:22 PM, harish singh <ha...@gmail.com>
> wrote:
>
> > This is how we have been doing it.
> >
> > pool_name = "pool_A"
> >
> > num_slots = 10
> > # creating pool
> > session = airflow.settings.Session()
> > pool = (
> > session.query(Pool)
> > .filter(Pool.pool == pool_name)
> > .first())
> > if not pool:
> > logging.info("Creating pool: {} with slots:
> > ".format(pool_name, str(num_slots)))
> > session.add(Pool(pool=pool_name, slots=num_slots))
> > session.commit()
> > session.close()
> >
> >
> >
> > On Tue, Sep 6, 2016 at 12:18 PM, Lance Norskog <la...@gmail.com>
> > wrote:
> >
> > > In the web UI, Admin->Pools lets you create, edit & view all Pools.
> > >
> > > On Sun, Sep 4, 2016 at 9:40 PM, הילה ויזן <hi...@gmail.com> wrote:
> > >
> > > > Hi,
> > > > we use a pool of size 1 to restrict parallelism of a task in dag.
> > > > We created the pool via airflow webserver.
> > > >
> > > > Is there any other way to create it via a script? maybe sql query on
> > DB?
> > > > I didn't find an appropriate CLI command.
> > > >
> > > > Thanks,
> > > > Hila
> > > >
> > >
> > >
> > >
> > > --
> > > Lance Norskog
> > > lance.norskog@gmail.com
> > > Redwood City, CA
> > >
> >
>
Re: creating a pool automatically
Posted by siddharth anand <sa...@apache.org>.
You can now create/modify pools via the Airflow CLI.
https://github.com/apache/incubator-airflow/pull/1735/commits
-s
Resources that are typically modified via the webserver, such as pools,
connections, & variables, should eventually be managed via the CLI to allow
for better integration with automation used to provision systems or deploy
new dags. Variables and pools (and I believe connections) are all done via
the CLI.
On Tue, Sep 6, 2016 at 12:22 PM, harish singh <ha...@gmail.com>
wrote:
> This is how we have been doing it.
>
> pool_name = "pool_A"
>
> num_slots = 10
> # creating pool
> session = airflow.settings.Session()
> pool = (
> session.query(Pool)
> .filter(Pool.pool == pool_name)
> .first())
> if not pool:
> logging.info("Creating pool: {} with slots:
> ".format(pool_name, str(num_slots)))
> session.add(Pool(pool=pool_name, slots=num_slots))
> session.commit()
> session.close()
>
>
>
> On Tue, Sep 6, 2016 at 12:18 PM, Lance Norskog <la...@gmail.com>
> wrote:
>
> > In the web UI, Admin->Pools lets you create, edit & view all Pools.
> >
> > On Sun, Sep 4, 2016 at 9:40 PM, הילה ויזן <hi...@gmail.com> wrote:
> >
> > > Hi,
> > > we use a pool of size 1 to restrict parallelism of a task in dag.
> > > We created the pool via airflow webserver.
> > >
> > > Is there any other way to create it via a script? maybe sql query on
> DB?
> > > I didn't find an appropriate CLI command.
> > >
> > > Thanks,
> > > Hila
> > >
> >
> >
> >
> > --
> > Lance Norskog
> > lance.norskog@gmail.com
> > Redwood City, CA
> >
>
Re: creating a pool automatically
Posted by harish singh <ha...@gmail.com>.
This is how we have been doing it.
pool_name = "pool_A"
num_slots = 10
# creating pool
session = airflow.settings.Session()
pool = (
session.query(Pool)
.filter(Pool.pool == pool_name)
.first())
if not pool:
logging.info("Creating pool: {} with slots:
".format(pool_name, str(num_slots)))
session.add(Pool(pool=pool_name, slots=num_slots))
session.commit()
session.close()
On Tue, Sep 6, 2016 at 12:18 PM, Lance Norskog <la...@gmail.com>
wrote:
> In the web UI, Admin->Pools lets you create, edit & view all Pools.
>
> On Sun, Sep 4, 2016 at 9:40 PM, הילה ויזן <hi...@gmail.com> wrote:
>
> > Hi,
> > we use a pool of size 1 to restrict parallelism of a task in dag.
> > We created the pool via airflow webserver.
> >
> > Is there any other way to create it via a script? maybe sql query on DB?
> > I didn't find an appropriate CLI command.
> >
> > Thanks,
> > Hila
> >
>
>
>
> --
> Lance Norskog
> lance.norskog@gmail.com
> Redwood City, CA
>
Re: creating a pool automatically
Posted by Lance Norskog <la...@gmail.com>.
In the web UI, Admin->Pools lets you create, edit & view all Pools.
On Sun, Sep 4, 2016 at 9:40 PM, הילה ויזן <hi...@gmail.com> wrote:
> Hi,
> we use a pool of size 1 to restrict parallelism of a task in dag.
> We created the pool via airflow webserver.
>
> Is there any other way to create it via a script? maybe sql query on DB?
> I didn't find an appropriate CLI command.
>
> Thanks,
> Hila
>
--
Lance Norskog
lance.norskog@gmail.com
Redwood City, CA