You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Tyrone Hinderson <th...@reonomy.com> on 2016/07/01 14:39:56 UTC
Re: Airflow pause-all
Interesting, is that config variable in the "core" section?
And environment variable config is still a thing in 1.7.1.2, if you're
interested.
On Thu, Jun 30, 2016 at 5:08 PM Laura Lorenz <ll...@industrydive.com>
wrote:
> Possibly useful to you is the variable dags_are_paused_at_creation = True
> you
> can set in airflow.cfg or in environment variables to cause the metadata
> database to consider all DAGs paused when they are first started up.
>
> I do think I saw something fly by on this dev list that environment
> variables for config got written out in the newest version. Not sure if
> it's true as I haven't tried to upgrade from 1.7.0 yet... partially for
> fear of tearing out our dependence on being able to control other airflow
> config with environment variables :)
>
> On Thu, Jun 30, 2016 at 4:59 PM, Tyrone Hinderson <th...@reonomy.com>
> wrote:
>
> > Hey there Chris,
> >
> > I agree that's a solid use case--mine is more development-oriented.
> Rather
> > than have all DAGs attempt to run upon scheduler start, a developer
> > probably just wants to observe how one DAG behaves. To that end, a
> > scheduler startup script can simply pause all DAGs before starting the
> dev
> > instance of the scheduler; then the developer can selectively unpause
> that
> > DAG via the UI or CLI (assuming those two controls are related).
> >
> > On Thu, Jun 30, 2016 at 4:06 PM Chris Riccomini <cr...@apache.org>
> > wrote:
> >
> > > Hey Tyrone,
> > >
> > > This would be a useful feature for us as well. Right now, we just turn
> > the
> > > scheduler off, which is kind of annoying.
> > >
> > > Use case is when things go really haywire (e.g. 100s of failures per
> > > minute), we just want everything to stop.
> > >
> > > Cheers,
> > > Chris
> > >
> > > On Thu, Jun 30, 2016 at 1:02 PM, Tyrone Hinderson <
> > thinderson@reonomy.com>
> > > wrote:
> > >
> > > > Hi, is there a way to "pause" all DAGs via the cli or otherwise?
> > > >
> > > > In the same vein, does "airflow pause" flip the same state which is
> > > flipped
> > > > by the on/off button next to each DAG in the UI?
> > > >
> > >
> >
>
Re: Airflow pause-all
Posted by Tyrone Hinderson <th...@reonomy.com>.
Cool, thanks for the clarification
On Fri, Jul 1, 2016 at 11:11 Laura Lorenz <ll...@industrydive.com> wrote:
> Before a given DAG has a history; if the metadata database has it listed as
> unpaused, the scheduler will honor that.
>
> On Fri, Jul 1, 2016 at 11:05 AM, Tyrone Hinderson <th...@reonomy.com>
> wrote:
>
> > I found dags_are_paused_at_creation, but what does "at creation" really
> > mean? Every time the scheduler starts, or before a given DAG has any
> > history?
> >
> > On Fri, Jul 1, 2016 at 10:39 AM Tyrone Hinderson <thinderson@reonomy.com
> >
> > wrote:
> >
> > > Interesting, is that config variable in the "core" section?
> > >
> > > And environment variable config is still a thing in 1.7.1.2, if you're
> > > interested.
> > >
> > > On Thu, Jun 30, 2016 at 5:08 PM Laura Lorenz <llorenz@industrydive.com
> >
> > > wrote:
> > >
> > >> Possibly useful to you is the variable dags_are_paused_at_creation =
> > True
> > >> you
> > >> can set in airflow.cfg or in environment variables to cause the
> metadata
> > >> database to consider all DAGs paused when they are first started up.
> > >>
> > >> I do think I saw something fly by on this dev list that environment
> > >> variables for config got written out in the newest version. Not sure
> if
> > >> it's true as I haven't tried to upgrade from 1.7.0 yet... partially
> for
> > >> fear of tearing out our dependence on being able to control other
> > airflow
> > >> config with environment variables :)
> > >>
> > >> On Thu, Jun 30, 2016 at 4:59 PM, Tyrone Hinderson <
> > thinderson@reonomy.com
> > >> >
> > >> wrote:
> > >>
> > >> > Hey there Chris,
> > >> >
> > >> > I agree that's a solid use case--mine is more development-oriented.
> > >> Rather
> > >> > than have all DAGs attempt to run upon scheduler start, a developer
> > >> > probably just wants to observe how one DAG behaves. To that end, a
> > >> > scheduler startup script can simply pause all DAGs before starting
> the
> > >> dev
> > >> > instance of the scheduler; then the developer can selectively
> unpause
> > >> that
> > >> > DAG via the UI or CLI (assuming those two controls are related).
> > >> >
> > >> > On Thu, Jun 30, 2016 at 4:06 PM Chris Riccomini <
> > criccomini@apache.org>
> > >> > wrote:
> > >> >
> > >> > > Hey Tyrone,
> > >> > >
> > >> > > This would be a useful feature for us as well. Right now, we just
> > turn
> > >> > the
> > >> > > scheduler off, which is kind of annoying.
> > >> > >
> > >> > > Use case is when things go really haywire (e.g. 100s of failures
> per
> > >> > > minute), we just want everything to stop.
> > >> > >
> > >> > > Cheers,
> > >> > > Chris
> > >> > >
> > >> > > On Thu, Jun 30, 2016 at 1:02 PM, Tyrone Hinderson <
> > >> > thinderson@reonomy.com>
> > >> > > wrote:
> > >> > >
> > >> > > > Hi, is there a way to "pause" all DAGs via the cli or otherwise?
> > >> > > >
> > >> > > > In the same vein, does "airflow pause" flip the same state which
> > is
> > >> > > flipped
> > >> > > > by the on/off button next to each DAG in the UI?
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>
Re: Airflow pause-all
Posted by Laura Lorenz <ll...@industrydive.com>.
Before a given DAG has a history; if the metadata database has it listed as
unpaused, the scheduler will honor that.
On Fri, Jul 1, 2016 at 11:05 AM, Tyrone Hinderson <th...@reonomy.com>
wrote:
> I found dags_are_paused_at_creation, but what does "at creation" really
> mean? Every time the scheduler starts, or before a given DAG has any
> history?
>
> On Fri, Jul 1, 2016 at 10:39 AM Tyrone Hinderson <th...@reonomy.com>
> wrote:
>
> > Interesting, is that config variable in the "core" section?
> >
> > And environment variable config is still a thing in 1.7.1.2, if you're
> > interested.
> >
> > On Thu, Jun 30, 2016 at 5:08 PM Laura Lorenz <ll...@industrydive.com>
> > wrote:
> >
> >> Possibly useful to you is the variable dags_are_paused_at_creation =
> True
> >> you
> >> can set in airflow.cfg or in environment variables to cause the metadata
> >> database to consider all DAGs paused when they are first started up.
> >>
> >> I do think I saw something fly by on this dev list that environment
> >> variables for config got written out in the newest version. Not sure if
> >> it's true as I haven't tried to upgrade from 1.7.0 yet... partially for
> >> fear of tearing out our dependence on being able to control other
> airflow
> >> config with environment variables :)
> >>
> >> On Thu, Jun 30, 2016 at 4:59 PM, Tyrone Hinderson <
> thinderson@reonomy.com
> >> >
> >> wrote:
> >>
> >> > Hey there Chris,
> >> >
> >> > I agree that's a solid use case--mine is more development-oriented.
> >> Rather
> >> > than have all DAGs attempt to run upon scheduler start, a developer
> >> > probably just wants to observe how one DAG behaves. To that end, a
> >> > scheduler startup script can simply pause all DAGs before starting the
> >> dev
> >> > instance of the scheduler; then the developer can selectively unpause
> >> that
> >> > DAG via the UI or CLI (assuming those two controls are related).
> >> >
> >> > On Thu, Jun 30, 2016 at 4:06 PM Chris Riccomini <
> criccomini@apache.org>
> >> > wrote:
> >> >
> >> > > Hey Tyrone,
> >> > >
> >> > > This would be a useful feature for us as well. Right now, we just
> turn
> >> > the
> >> > > scheduler off, which is kind of annoying.
> >> > >
> >> > > Use case is when things go really haywire (e.g. 100s of failures per
> >> > > minute), we just want everything to stop.
> >> > >
> >> > > Cheers,
> >> > > Chris
> >> > >
> >> > > On Thu, Jun 30, 2016 at 1:02 PM, Tyrone Hinderson <
> >> > thinderson@reonomy.com>
> >> > > wrote:
> >> > >
> >> > > > Hi, is there a way to "pause" all DAGs via the cli or otherwise?
> >> > > >
> >> > > > In the same vein, does "airflow pause" flip the same state which
> is
> >> > > flipped
> >> > > > by the on/off button next to each DAG in the UI?
> >> > > >
> >> > >
> >> >
> >>
> >
>
Re: Airflow pause-all
Posted by Tyrone Hinderson <th...@reonomy.com>.
I found dags_are_paused_at_creation, but what does "at creation" really
mean? Every time the scheduler starts, or before a given DAG has any
history?
On Fri, Jul 1, 2016 at 10:39 AM Tyrone Hinderson <th...@reonomy.com>
wrote:
> Interesting, is that config variable in the "core" section?
>
> And environment variable config is still a thing in 1.7.1.2, if you're
> interested.
>
> On Thu, Jun 30, 2016 at 5:08 PM Laura Lorenz <ll...@industrydive.com>
> wrote:
>
>> Possibly useful to you is the variable dags_are_paused_at_creation = True
>> you
>> can set in airflow.cfg or in environment variables to cause the metadata
>> database to consider all DAGs paused when they are first started up.
>>
>> I do think I saw something fly by on this dev list that environment
>> variables for config got written out in the newest version. Not sure if
>> it's true as I haven't tried to upgrade from 1.7.0 yet... partially for
>> fear of tearing out our dependence on being able to control other airflow
>> config with environment variables :)
>>
>> On Thu, Jun 30, 2016 at 4:59 PM, Tyrone Hinderson <thinderson@reonomy.com
>> >
>> wrote:
>>
>> > Hey there Chris,
>> >
>> > I agree that's a solid use case--mine is more development-oriented.
>> Rather
>> > than have all DAGs attempt to run upon scheduler start, a developer
>> > probably just wants to observe how one DAG behaves. To that end, a
>> > scheduler startup script can simply pause all DAGs before starting the
>> dev
>> > instance of the scheduler; then the developer can selectively unpause
>> that
>> > DAG via the UI or CLI (assuming those two controls are related).
>> >
>> > On Thu, Jun 30, 2016 at 4:06 PM Chris Riccomini <cr...@apache.org>
>> > wrote:
>> >
>> > > Hey Tyrone,
>> > >
>> > > This would be a useful feature for us as well. Right now, we just turn
>> > the
>> > > scheduler off, which is kind of annoying.
>> > >
>> > > Use case is when things go really haywire (e.g. 100s of failures per
>> > > minute), we just want everything to stop.
>> > >
>> > > Cheers,
>> > > Chris
>> > >
>> > > On Thu, Jun 30, 2016 at 1:02 PM, Tyrone Hinderson <
>> > thinderson@reonomy.com>
>> > > wrote:
>> > >
>> > > > Hi, is there a way to "pause" all DAGs via the cli or otherwise?
>> > > >
>> > > > In the same vein, does "airflow pause" flip the same state which is
>> > > flipped
>> > > > by the on/off button next to each DAG in the UI?
>> > > >
>> > >
>> >
>>
>
Re: Airflow pause-all
Posted by Laura Lorenz <ll...@industrydive.com>.
Yes, it is in the core section. Thanks for the clarification on 1.7.1.2!
On Fri, Jul 1, 2016 at 10:39 AM, Tyrone Hinderson <th...@reonomy.com>
wrote:
> Interesting, is that config variable in the "core" section?
>
> And environment variable config is still a thing in 1.7.1.2, if you're
> interested.
>
> On Thu, Jun 30, 2016 at 5:08 PM Laura Lorenz <ll...@industrydive.com>
> wrote:
>
> > Possibly useful to you is the variable dags_are_paused_at_creation = True
> > you
> > can set in airflow.cfg or in environment variables to cause the metadata
> > database to consider all DAGs paused when they are first started up.
> >
> > I do think I saw something fly by on this dev list that environment
> > variables for config got written out in the newest version. Not sure if
> > it's true as I haven't tried to upgrade from 1.7.0 yet... partially for
> > fear of tearing out our dependence on being able to control other airflow
> > config with environment variables :)
> >
> > On Thu, Jun 30, 2016 at 4:59 PM, Tyrone Hinderson <
> thinderson@reonomy.com>
> > wrote:
> >
> > > Hey there Chris,
> > >
> > > I agree that's a solid use case--mine is more development-oriented.
> > Rather
> > > than have all DAGs attempt to run upon scheduler start, a developer
> > > probably just wants to observe how one DAG behaves. To that end, a
> > > scheduler startup script can simply pause all DAGs before starting the
> > dev
> > > instance of the scheduler; then the developer can selectively unpause
> > that
> > > DAG via the UI or CLI (assuming those two controls are related).
> > >
> > > On Thu, Jun 30, 2016 at 4:06 PM Chris Riccomini <criccomini@apache.org
> >
> > > wrote:
> > >
> > > > Hey Tyrone,
> > > >
> > > > This would be a useful feature for us as well. Right now, we just
> turn
> > > the
> > > > scheduler off, which is kind of annoying.
> > > >
> > > > Use case is when things go really haywire (e.g. 100s of failures per
> > > > minute), we just want everything to stop.
> > > >
> > > > Cheers,
> > > > Chris
> > > >
> > > > On Thu, Jun 30, 2016 at 1:02 PM, Tyrone Hinderson <
> > > thinderson@reonomy.com>
> > > > wrote:
> > > >
> > > > > Hi, is there a way to "pause" all DAGs via the cli or otherwise?
> > > > >
> > > > > In the same vein, does "airflow pause" flip the same state which is
> > > > flipped
> > > > > by the on/off button next to each DAG in the UI?
> > > > >
> > > >
> > >
> >
>