You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airflow.apache.org by Shah Altaf <me...@gmail.com> on 2018/10/07 18:31:19 UTC
Can a DAG be conditionally hidden from the UI?
Hi all,
tl;dr - Is it possible to conditionally hide a DAG from the UI based on an
environment variable?
Our team has a single repo with several DAGs in it and we deploy it across
multiple 'environments' (think dev, test, and other integration
environments). While most DAGs are meant to run everywhere, we do have
several which are meant to run in one and only one environment. Of course
they are all paused, but it would be nice to declutter a bit for ourselves.
My question then - is it possible to conditionally hide a DAG from the UI
based on an environment variable or some flag somewhere.
This is just wishful thinking - the dev could do something like
dag = get_dag(...),and get_dag() would have a decorator like
@only_run_in("integration4,dev,local")
And that decorator returns some kind of None object or special DAG which
just doesn't appear in the list.
Or perhaps some other way to accomplish this same effect - any ideas would
be appreciated.
Thanks
Re: Can a DAG be conditionally hidden from the UI?
Posted by James Meickle <jm...@quantopian.com.INVALID>.
It's fine (and required if you do task assignment) to create the DAG
object; just return None from the function (return early) so that there is
no variable containing a DAG object visible within the module.
On Tue, Oct 9, 2018 at 12:21 PM Shah Altaf <me...@gmail.com> wrote:
> @*jmeickle *can you share your code or any sample? How are you returning
> nothing?
>
> I may have spoken too soon. I can't seem to set dag to None, it fails with
>
> *>>>>Tried to create relationships between tasks that don't have DAGs yet*
>
>
> On Tue, Oct 9, 2018 at 12:57 PM Shah Altaf <me...@gmail.com> wrote:
>
> >
> >
> > Thanks all, following your advice I've decided to do this right now with
> a
> > decorator. It looks crude so any suggestions are welcome.
> >
> > The decorator function checks an environment variable that we set on our
> > servers and if it matches any of the passed in environment names, it'll
> > allow the DAG to be created.
> >
> >
> > def environments(env_names):
> > def restrict_environment(func):
> > def func_wrapper(*args, **kwargs):
> > * if( os.getenv("environment") in env_names):*
> > return DAG(*args, **kwargs)
> > return func_wrapper
> > return restrict_environment
> >
> > *@environments(["env1","env2","production"])*
> > def get_DAG(*args, **kwargs):
> > return None
> >
> > dag = get_DAG('my_dag', default_args=default_args, schedule_interval="30
> 5
> > * * *")
> >
> >
> > Regards
> > Shah
> >
> >
> > On Mon, Oct 8, 2018 at 5:19 PM William Pursell
> <wi...@wepay.com.invalid>
> > wrote:
> >
> >> This would be a very desirable feature. It's not just an issue of
> >> differing environments, but also changing requirements. A dag may be
> >> used for a time, and then removed from the environment. But if it's
> >> still in the airflow db, it will get a row on the UI with a black "I'
> >> indicating that the dag is missing. As far as I know, then only way
> >> to remove it is to manually edit the database.
> >> On Mon, Oct 8, 2018 at 9:43 AM Chris Palmer <ch...@crpalmer.com> wrote:
> >> >
> >> > I like James solution better, but the initial thought I had was to
> >> deploy
> >> > airflowignore files to the environments to filter out files that
> should
> >> not
> >> > be processed when filling the DagBag.
> >> >
> >> > Chris
> >> >
> >> > On Mon, Oct 8, 2018 at 10:22 AM James Meickle
> >> > <jm...@quantopian.com.invalid> wrote:
> >> >
> >> > > As long as the Airflow process can't find the DAG as a top-level
> >> object in
> >> > > the module, it won't be registered. For example, we have a function
> >> that
> >> > > returns DAGs; the function returns nothing if it's not in the right
> >> > > environment.
> >> > >
> >> > > On Sun, Oct 7, 2018 at 2:31 PM Shah Altaf <me...@gmail.com>
> wrote:
> >> > >
> >> > > > Hi all,
> >> > > >
> >> > > > tl;dr - Is it possible to conditionally hide a DAG from the UI
> >> based on
> >> > > an
> >> > > > environment variable?
> >> > > >
> >> > > >
> >> > > >
> >> > > > Our team has a single repo with several DAGs in it and we deploy
> it
> >> > > across
> >> > > > multiple 'environments' (think dev, test, and other integration
> >> > > > environments). While most DAGs are meant to run everywhere, we do
> >> have
> >> > > > several which are meant to run in one and only one environment.
> Of
> >> > > course
> >> > > > they are all paused, but it would be nice to declutter a bit for
> >> > > ourselves.
> >> > > >
> >> > > > My question then - is it possible to conditionally hide a DAG from
> >> the UI
> >> > > > based on an environment variable or some flag somewhere.
> >> > > >
> >> > > > This is just wishful thinking - the dev could do something like
> >> > > >
> >> > > > dag = get_dag(...),and get_dag() would have a decorator like
> >> > > > @only_run_in("integration4,dev,local")
> >> > > >
> >> > > > And that decorator returns some kind of None object or special DAG
> >> which
> >> > > > just doesn't appear in the list.
> >> > > >
> >> > > > Or perhaps some other way to accomplish this same effect - any
> ideas
> >> > > would
> >> > > > be appreciated.
> >> > > >
> >> > > > Thanks
> >> > > >
> >> > >
> >>
> >>
>
Re: Can a DAG be conditionally hidden from the UI?
Posted by Shah Altaf <me...@gmail.com>.
@*jmeickle *can you share your code or any sample? How are you returning
nothing?
I may have spoken too soon. I can't seem to set dag to None, it fails with
*>>>>Tried to create relationships between tasks that don't have DAGs yet*
On Tue, Oct 9, 2018 at 12:57 PM Shah Altaf <me...@gmail.com> wrote:
>
>
> Thanks all, following your advice I've decided to do this right now with a
> decorator. It looks crude so any suggestions are welcome.
>
> The decorator function checks an environment variable that we set on our
> servers and if it matches any of the passed in environment names, it'll
> allow the DAG to be created.
>
>
> def environments(env_names):
> def restrict_environment(func):
> def func_wrapper(*args, **kwargs):
> * if( os.getenv("environment") in env_names):*
> return DAG(*args, **kwargs)
> return func_wrapper
> return restrict_environment
>
> *@environments(["env1","env2","production"])*
> def get_DAG(*args, **kwargs):
> return None
>
> dag = get_DAG('my_dag', default_args=default_args, schedule_interval="30 5
> * * *")
>
>
> Regards
> Shah
>
>
> On Mon, Oct 8, 2018 at 5:19 PM William Pursell <wi...@wepay.com.invalid>
> wrote:
>
>> This would be a very desirable feature. It's not just an issue of
>> differing environments, but also changing requirements. A dag may be
>> used for a time, and then removed from the environment. But if it's
>> still in the airflow db, it will get a row on the UI with a black "I'
>> indicating that the dag is missing. As far as I know, then only way
>> to remove it is to manually edit the database.
>> On Mon, Oct 8, 2018 at 9:43 AM Chris Palmer <ch...@crpalmer.com> wrote:
>> >
>> > I like James solution better, but the initial thought I had was to
>> deploy
>> > airflowignore files to the environments to filter out files that should
>> not
>> > be processed when filling the DagBag.
>> >
>> > Chris
>> >
>> > On Mon, Oct 8, 2018 at 10:22 AM James Meickle
>> > <jm...@quantopian.com.invalid> wrote:
>> >
>> > > As long as the Airflow process can't find the DAG as a top-level
>> object in
>> > > the module, it won't be registered. For example, we have a function
>> that
>> > > returns DAGs; the function returns nothing if it's not in the right
>> > > environment.
>> > >
>> > > On Sun, Oct 7, 2018 at 2:31 PM Shah Altaf <me...@gmail.com> wrote:
>> > >
>> > > > Hi all,
>> > > >
>> > > > tl;dr - Is it possible to conditionally hide a DAG from the UI
>> based on
>> > > an
>> > > > environment variable?
>> > > >
>> > > >
>> > > >
>> > > > Our team has a single repo with several DAGs in it and we deploy it
>> > > across
>> > > > multiple 'environments' (think dev, test, and other integration
>> > > > environments). While most DAGs are meant to run everywhere, we do
>> have
>> > > > several which are meant to run in one and only one environment. Of
>> > > course
>> > > > they are all paused, but it would be nice to declutter a bit for
>> > > ourselves.
>> > > >
>> > > > My question then - is it possible to conditionally hide a DAG from
>> the UI
>> > > > based on an environment variable or some flag somewhere.
>> > > >
>> > > > This is just wishful thinking - the dev could do something like
>> > > >
>> > > > dag = get_dag(...),and get_dag() would have a decorator like
>> > > > @only_run_in("integration4,dev,local")
>> > > >
>> > > > And that decorator returns some kind of None object or special DAG
>> which
>> > > > just doesn't appear in the list.
>> > > >
>> > > > Or perhaps some other way to accomplish this same effect - any ideas
>> > > would
>> > > > be appreciated.
>> > > >
>> > > > Thanks
>> > > >
>> > >
>>
>>
Re: Can a DAG be conditionally hidden from the UI?
Posted by Shah Altaf <me...@gmail.com>.
Thanks all, following your advice I've decided to do this right now with a
decorator. It looks crude so any suggestions are welcome.
The decorator function checks an environment variable that we set on our
servers and if it matches any of the passed in environment names, it'll
allow the DAG to be created.
def environments(env_names):
def restrict_environment(func):
def func_wrapper(*args, **kwargs):
* if( os.getenv("environment") in env_names):*
return DAG(*args, **kwargs)
return func_wrapper
return restrict_environment
*@environments(["env1","env2","production"])*
def get_DAG(*args, **kwargs):
return None
dag = get_DAG('my_dag', default_args=default_args, schedule_interval="30 5
* * *")
Regards
Shah
On Mon, Oct 8, 2018 at 5:19 PM William Pursell <wi...@wepay.com.invalid>
wrote:
> This would be a very desirable feature. It's not just an issue of
> differing environments, but also changing requirements. A dag may be
> used for a time, and then removed from the environment. But if it's
> still in the airflow db, it will get a row on the UI with a black "I'
> indicating that the dag is missing. As far as I know, then only way
> to remove it is to manually edit the database.
> On Mon, Oct 8, 2018 at 9:43 AM Chris Palmer <ch...@crpalmer.com> wrote:
> >
> > I like James solution better, but the initial thought I had was to deploy
> > airflowignore files to the environments to filter out files that should
> not
> > be processed when filling the DagBag.
> >
> > Chris
> >
> > On Mon, Oct 8, 2018 at 10:22 AM James Meickle
> > <jm...@quantopian.com.invalid> wrote:
> >
> > > As long as the Airflow process can't find the DAG as a top-level
> object in
> > > the module, it won't be registered. For example, we have a function
> that
> > > returns DAGs; the function returns nothing if it's not in the right
> > > environment.
> > >
> > > On Sun, Oct 7, 2018 at 2:31 PM Shah Altaf <me...@gmail.com> wrote:
> > >
> > > > Hi all,
> > > >
> > > > tl;dr - Is it possible to conditionally hide a DAG from the UI based
> on
> > > an
> > > > environment variable?
> > > >
> > > >
> > > >
> > > > Our team has a single repo with several DAGs in it and we deploy it
> > > across
> > > > multiple 'environments' (think dev, test, and other integration
> > > > environments). While most DAGs are meant to run everywhere, we do
> have
> > > > several which are meant to run in one and only one environment. Of
> > > course
> > > > they are all paused, but it would be nice to declutter a bit for
> > > ourselves.
> > > >
> > > > My question then - is it possible to conditionally hide a DAG from
> the UI
> > > > based on an environment variable or some flag somewhere.
> > > >
> > > > This is just wishful thinking - the dev could do something like
> > > >
> > > > dag = get_dag(...),and get_dag() would have a decorator like
> > > > @only_run_in("integration4,dev,local")
> > > >
> > > > And that decorator returns some kind of None object or special DAG
> which
> > > > just doesn't appear in the list.
> > > >
> > > > Or perhaps some other way to accomplish this same effect - any ideas
> > > would
> > > > be appreciated.
> > > >
> > > > Thanks
> > > >
> > >
>
>
Re: Can a DAG be conditionally hidden from the UI?
Posted by William Pursell <wi...@wepay.com.INVALID>.
This would be a very desirable feature. It's not just an issue of
differing environments, but also changing requirements. A dag may be
used for a time, and then removed from the environment. But if it's
still in the airflow db, it will get a row on the UI with a black "I'
indicating that the dag is missing. As far as I know, then only way
to remove it is to manually edit the database.
On Mon, Oct 8, 2018 at 9:43 AM Chris Palmer <ch...@crpalmer.com> wrote:
>
> I like James solution better, but the initial thought I had was to deploy
> airflowignore files to the environments to filter out files that should not
> be processed when filling the DagBag.
>
> Chris
>
> On Mon, Oct 8, 2018 at 10:22 AM James Meickle
> <jm...@quantopian.com.invalid> wrote:
>
> > As long as the Airflow process can't find the DAG as a top-level object in
> > the module, it won't be registered. For example, we have a function that
> > returns DAGs; the function returns nothing if it's not in the right
> > environment.
> >
> > On Sun, Oct 7, 2018 at 2:31 PM Shah Altaf <me...@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > tl;dr - Is it possible to conditionally hide a DAG from the UI based on
> > an
> > > environment variable?
> > >
> > >
> > >
> > > Our team has a single repo with several DAGs in it and we deploy it
> > across
> > > multiple 'environments' (think dev, test, and other integration
> > > environments). While most DAGs are meant to run everywhere, we do have
> > > several which are meant to run in one and only one environment. Of
> > course
> > > they are all paused, but it would be nice to declutter a bit for
> > ourselves.
> > >
> > > My question then - is it possible to conditionally hide a DAG from the UI
> > > based on an environment variable or some flag somewhere.
> > >
> > > This is just wishful thinking - the dev could do something like
> > >
> > > dag = get_dag(...),and get_dag() would have a decorator like
> > > @only_run_in("integration4,dev,local")
> > >
> > > And that decorator returns some kind of None object or special DAG which
> > > just doesn't appear in the list.
> > >
> > > Or perhaps some other way to accomplish this same effect - any ideas
> > would
> > > be appreciated.
> > >
> > > Thanks
> > >
> >
Re: Can a DAG be conditionally hidden from the UI?
Posted by Chris Palmer <ch...@crpalmer.com>.
I like James solution better, but the initial thought I had was to deploy
airflowignore files to the environments to filter out files that should not
be processed when filling the DagBag.
Chris
On Mon, Oct 8, 2018 at 10:22 AM James Meickle
<jm...@quantopian.com.invalid> wrote:
> As long as the Airflow process can't find the DAG as a top-level object in
> the module, it won't be registered. For example, we have a function that
> returns DAGs; the function returns nothing if it's not in the right
> environment.
>
> On Sun, Oct 7, 2018 at 2:31 PM Shah Altaf <me...@gmail.com> wrote:
>
> > Hi all,
> >
> > tl;dr - Is it possible to conditionally hide a DAG from the UI based on
> an
> > environment variable?
> >
> >
> >
> > Our team has a single repo with several DAGs in it and we deploy it
> across
> > multiple 'environments' (think dev, test, and other integration
> > environments). While most DAGs are meant to run everywhere, we do have
> > several which are meant to run in one and only one environment. Of
> course
> > they are all paused, but it would be nice to declutter a bit for
> ourselves.
> >
> > My question then - is it possible to conditionally hide a DAG from the UI
> > based on an environment variable or some flag somewhere.
> >
> > This is just wishful thinking - the dev could do something like
> >
> > dag = get_dag(...),and get_dag() would have a decorator like
> > @only_run_in("integration4,dev,local")
> >
> > And that decorator returns some kind of None object or special DAG which
> > just doesn't appear in the list.
> >
> > Or perhaps some other way to accomplish this same effect - any ideas
> would
> > be appreciated.
> >
> > Thanks
> >
>
Re: Can a DAG be conditionally hidden from the UI?
Posted by James Meickle <jm...@quantopian.com.INVALID>.
As long as the Airflow process can't find the DAG as a top-level object in
the module, it won't be registered. For example, we have a function that
returns DAGs; the function returns nothing if it's not in the right
environment.
On Sun, Oct 7, 2018 at 2:31 PM Shah Altaf <me...@gmail.com> wrote:
> Hi all,
>
> tl;dr - Is it possible to conditionally hide a DAG from the UI based on an
> environment variable?
>
>
>
> Our team has a single repo with several DAGs in it and we deploy it across
> multiple 'environments' (think dev, test, and other integration
> environments). While most DAGs are meant to run everywhere, we do have
> several which are meant to run in one and only one environment. Of course
> they are all paused, but it would be nice to declutter a bit for ourselves.
>
> My question then - is it possible to conditionally hide a DAG from the UI
> based on an environment variable or some flag somewhere.
>
> This is just wishful thinking - the dev could do something like
>
> dag = get_dag(...),and get_dag() would have a decorator like
> @only_run_in("integration4,dev,local")
>
> And that decorator returns some kind of None object or special DAG which
> just doesn't appear in the list.
>
> Or perhaps some other way to accomplish this same effect - any ideas would
> be appreciated.
>
> Thanks
>